Sep 15 21:06:23.980512 ip-10-0-141-188 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 21:06:23.980525 ip-10-0-141-188 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 21:06:23.980532 ip-10-0-141-188 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 21:06:23.980705 ip-10-0-141-188 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 21:06:34.137734 ip-10-0-141-188 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 21:06:34.137750 ip-10-0-141-188 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8a2014971b2343d2802a318ec353dfbc --
Sep 15 21:08:21.076541 ip-10-0-141-188 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 21:08:21.503533 ip-10-0-141-188 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:08:21.503533 ip-10-0-141-188 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 21:08:21.503533 ip-10-0-141-188 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:08:21.503533 ip-10-0-141-188 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 21:08:21.503533 ip-10-0-141-188 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:08:21.504439 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.504350    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 21:08:21.508856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508830    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:21.508856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508852    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:21.508856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508856    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:21.508856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508860    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508864    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508868    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508871    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508874    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508877    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508880    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508883    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508885    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508888    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508891    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508893    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508896    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508898    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508901    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508904    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508908    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508912    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508915    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:21.509036 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508918    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508921    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508924    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508926    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508929    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508932    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508934    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508937    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508940    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508942    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508951    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508954    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508957    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508959    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508962    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508965    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508968    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508971    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508974    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508976    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:21.509515 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508979    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508981    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508984    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508986    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508989    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508991    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508994    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508996    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.508999    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509001    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509004    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509007    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509009    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509012    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509014    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509017    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509019    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509023    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509025    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509028    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:21.509993 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509031    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509034    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509037    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509040    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509043    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509045    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509048    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509051    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509054    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509056    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509059    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509062    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509064    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509067    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509069    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509072    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509075    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509077    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509081    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509084    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:21.510502 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509086    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509090    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509093    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509097    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509100    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509103    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509106    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509108    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509111    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509113    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509116    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509118    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509122    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509126    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509129    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509131    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509134    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509884    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:21.511013 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509892    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509896    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509899    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509902    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509904    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509907    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509910    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509912    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509915    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509917    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509920    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509924    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509928    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509931    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509935    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509937    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509941    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509943    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509946    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509949    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:21.511464 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509951    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509954    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509956    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509959    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509961    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509963    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509966    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509969    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509971    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509973    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509976    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509978    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509982    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509984    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509988    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509993    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.509996    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510000    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510003    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:21.511953 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510006    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510009    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510011    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510015    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510018    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510020    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510023    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510025    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510028    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510031    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510034    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510036    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510040    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510042    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510045    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510047    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510050    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510052    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510055    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510057    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:21.512422 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510060    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510062    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510065    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510067    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510069    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510072    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510075    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510077    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510080    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510082    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510084    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510087    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510089    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510092    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510094    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510097    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510100    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510102    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510105    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510107    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:21.512905 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510110    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510113    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510116    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510120    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510122    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510125    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510127    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510130    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510132    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510135    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510137    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510140    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510142    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510145    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510148    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510150    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510153    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510155    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510158    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510546    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:21.513408 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510551    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510554    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510556    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510559    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510563    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510565    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510568    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510571    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510574    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510576    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510579    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510581    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510584    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510587    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510590    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510593    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510596    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510598    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510602    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:21.513894 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510604    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510607    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510609    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510613    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510617    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510620    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510622    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510625    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510628    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510630    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510633    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510636    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510639    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510642    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510648    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510651    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510655    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510658    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510661    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:21.514375 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510666    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510669    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510672    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510675    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510677    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510680    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510683    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510685    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510688    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510691    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510694    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510697    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510699    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510702    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510704    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510707    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510709    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510712    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510714    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510716    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:21.514839 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510719    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510721    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510723    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510728    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510731    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510733    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510735    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510739    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510742    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510744    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510747    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510749    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510752    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510754    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510756    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510759    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510762    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510764    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510767    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510769    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:21.515338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510772    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510774    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510776    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510779    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510782    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510784    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510786    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510789    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510791    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510794    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510796    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510799    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510803    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510805    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510808    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510811    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510814    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510816    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510819    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.510822    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:21.515856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510895    2551 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510933    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510938    2551 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510943    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510947    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510950    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510955    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510959    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510962    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510965    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510968    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510971    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510974    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510977    2551 flags.go:64] FLAG: --cgroup-root=""
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510980    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510983    2551 flags.go:64] FLAG: --client-ca-file=""
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510988    2551 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510991    2551 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510995    2551 flags.go:64] FLAG: --cluster-domain=""
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.510998    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511001    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511004    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511008    2551 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511011    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 21:08:21.516341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511014    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511017    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511021    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511026    2551 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511029    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511032    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511035    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511039    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511043    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511046    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511049    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511052    2551 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511054    2551 flags.go:64] FLAG: --enable-server="true"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511057    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511061    2551 flags.go:64] FLAG: --event-burst="100"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511065    2551 flags.go:64] FLAG: --event-qps="50"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511067    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511070    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511073    2551 flags.go:64] FLAG: --eviction-hard=""
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511081    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511084    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511087    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511090    2551 flags.go:64] FLAG: --eviction-soft=""
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511093    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511096    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 21:08:21.516923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511099    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511102    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511105    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511108    2551 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511111    2551 flags.go:64] FLAG: --feature-gates=""
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511115    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511118    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511121    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511124    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511127    2551 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511131    2551 flags.go:64] FLAG: --help="false"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511134    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511137    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511140    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511143    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511146    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511150    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511153    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511156    2551 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511158    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511161    2551 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511164    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511167    2551 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511170    2551 flags.go:64] FLAG: --kube-reserved=""
Sep 15 21:08:21.517546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511173    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511175    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511178    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511181    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511183    2551 flags.go:64] FLAG: --lock-file=""
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511186    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511204    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511207    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511213    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511217    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511220    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511223    2551 flags.go:64] FLAG: --logging-format="text"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511225    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511229    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511231    2551 flags.go:64] FLAG: --manifest-url=""
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511234    2551 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511239    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511242    2551 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511247    2551 flags.go:64] FLAG: --max-pods="110"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511250    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511253    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511255    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511258    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511261    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511264    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 21:08:21.518129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511267    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511274    2551 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511277    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511280    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511283    2551 flags.go:64] FLAG: --pod-cidr=""
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511286    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511291    2551 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511294    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511297    2551 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511299    2551 flags.go:64] FLAG: --port="10250"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511303    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511305    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc084e8ecf2b5754"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511309    2551 flags.go:64] FLAG: --qos-reserved=""
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511311    2551 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511314    2551 flags.go:64] FLAG: --register-node="true"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511318    2551 flags.go:64] FLAG: --register-with-taints=""
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511321    2551 flags.go:64] FLAG: --registry-burst="10"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511325    2551 flags.go:64] FLAG: --registry-qps="5"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511327    2551 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511330    2551 flags.go:64] FLAG: --reserved-memory=""
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511334    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511337    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511340    2551 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511343    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511345    2551 flags.go:64] FLAG: --runonce="false"
Sep 15 21:08:21.518799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511348    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511352    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511355    2551 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511358    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511361    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511364    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511367    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511370    2551 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511373    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511376    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511379    2551 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511382    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511384    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511387    2551 flags.go:64] FLAG: --system-cgroups=""
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511390    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511395    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511398    2551 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511400    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511404    2551 flags.go:64] FLAG: --tls-min-version=""
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511407    2551 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511410    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511414    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511416    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511420    2551 flags.go:64] FLAG: --v="2"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511424    2551 flags.go:64] FLAG: --version="false"
Sep 15 21:08:21.519560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511428    2551 flags.go:64] FLAG: --vmodule=""
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511432    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.511435    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511547    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511551    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511554    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511557    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511559    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511563    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511566    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511568    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511571    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511574    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511576    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511579    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511581    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511584    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511586    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511589    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511592    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:21.520414 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511594    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511597    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511601    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511604    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511609    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511612    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511616    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511619    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511624    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511626    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511629    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511631    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511634    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511637    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511640    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511643    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511647    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511651    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511654    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:21.521327 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511657    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511661    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511664    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511668    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511672    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511674    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511677    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511679    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511682    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511685    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511688    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511690    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511693    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511695    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511698    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511700    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511703    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511705    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:21.522147 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511708    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511710    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511713    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511716    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511719    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511721    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511724    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511726    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511729    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511733    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511736    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511739    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511741    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511744    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511746    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511750    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511753    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511756    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511758    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511762    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:21.522801 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511765    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511769    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511772    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511775    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511777    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511780    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511782    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511785    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511788    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511790    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511793    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511795    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511797    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511800    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511802    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511806    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511809    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511812    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511814    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511817    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:21.523398 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511819    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511821    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511825    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511827    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.511830    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.512390    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.522524    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.522546    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.523430    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.523446    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523535    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523543    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523548    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523553    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:21.524284 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523557    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523562    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523566    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523570    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523574    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523579    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523583    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523588    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523592    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523597    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523601    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523605    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523609    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523613    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523617    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523621    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523625    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523629    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523634    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523638    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:21.524930 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523643    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523647    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523652    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523659    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523665    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523673    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523679    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523684    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523689    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523694    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523698    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523702    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523706    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523710    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523715    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523719    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523723    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523727    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523732    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:21.525475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523736    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523740    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523744    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523748    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523752    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523756    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523760    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523764    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523768    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523772    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523776    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523780    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523784    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523788    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523793    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523797    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523801    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523805    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523811    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523816    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:21.526241 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523820    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523824    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523828    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523833    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523837    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523841    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523845    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523849    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523853    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523857    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523861    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523866    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523870    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523874    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523878    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523882    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523886    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523890    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523894    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523898    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:21.527063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523903    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523907    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523911    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523915    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523919    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523923    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523927    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523932    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523936    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523940    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523945    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523950    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523954    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523958    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523964    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:21.527589 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.523970    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.523978    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524149    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524155    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524160    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524165    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524169    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524173    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524177    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524182    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524186    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524210    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524215    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524219    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:08:21.528063 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524223    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524227    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524231    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524235    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524240    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524244    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524248    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524254    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524260    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524265    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524270    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524275    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524280    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524284    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524289    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524294    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524299    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524303    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524308    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:08:21.528475 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524313    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524317    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524322    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524326    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524330    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524334    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524338    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524342    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524347    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524351    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524354    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524359    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524363    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524367    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524371    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524375    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524379    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524384    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524388    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524392    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:08:21.528979 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524396    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524400    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524404    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524408    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524413    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524418    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524422    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524426    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524430    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524434    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524439    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524443    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524447    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524451    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524455    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524483    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524489    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524494    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524498    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:08:21.529553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524502    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524507    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524510    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524515    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524519    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524523    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524527    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524531    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524535    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524540    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524545    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524551    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524556    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524561    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524565    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524569    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524573    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524578    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524582    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524586    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:08:21.530006 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524591    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524598    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524603    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524610    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524615    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524620    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524625    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524629    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:21.524634    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.524641    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.524795    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.527646    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.528541    2551 server.go:1019] "Starting client certificate rotation"
Sep 15 21:08:21.530527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.528638    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:08:21.530909 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.529396    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:08:21.531265 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.531249    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:08:21.537786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.537633    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:08:21.548661 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.548643    2551 log.go:25] "Validated CRI v1 runtime API"
Sep 15 21:08:21.551848 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.551832    2551 log.go:25] "Validated CRI v1 image API"
Sep 15 21:08:21.554286 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.554264    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 21:08:21.556954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.556938    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:08:21.558092 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.558071    2551 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8b20a026-75f5-4ea5-9fd9-f7502515f2a8:/dev/nvme0n1p3 980b4836-03cb-4177-9682-4deef5783ae1:/dev/nvme0n1p4]
Sep 15 21:08:21.558169 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.558091    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 21:08:21.563157 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.563041    2551 manager.go:216] Machine: {Timestamp:2026-09-15 21:08:21.561748078 +0000 UTC m=+0.363617024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101288 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e3633905ec17050ab9c98b32e9b4 SystemUUID:ec29e363-3905-ec17-050a-b9c98b32e9b4 BootID:8a201497-1b23-43d2-802a-318ec353dfbc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:94:1c:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:94:1c:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:94:bc:94:11:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 21:08:21.563157 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.563148    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 21:08:21.563346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.563283    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 21:08:21.566517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.566482    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 21:08:21.566679 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.566521    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-188.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 21:08:21.567306 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.567294    2551 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 21:08:21.567368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.567311    2551 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 21:08:21.567368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.567321    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:08:21.568056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.568044    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:08:21.569246 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.569234    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 21:08:21.572512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572498    2551 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:08:21.572674 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572662    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 21:08:21.572737 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572700    2551 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 21:08:21.572737 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572724    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 21:08:21.572824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572745    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 21:08:21.572824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572803    2551 kubelet.go:398] "Adding apiserver pod source"
Sep 15 21:08:21.572824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.572817    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 21:08:21.573276 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.573262    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:08:21.573354 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.573284    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:08:21.576428 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.576403    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 21:08:21.579955 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.579936    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 21:08:21.580036 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.579969    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 21:08:21.583268 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583236    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 21:08:21.583414 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583387    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 21:08:21.583414 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583407    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583423    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583436    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583445    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583465    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583473    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583500    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583510    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583537    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 21:08:21.583567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.583546    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 21:08:21.584256 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.584230    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 21:08:21.584475 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.584440    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 21:08:21.584475 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.584473    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 21:08:21.584599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.584571    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 21:08:21.587680 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.587662    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-188.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 21:08:21.588948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.588932    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4t7vc"
Sep 15 21:08:21.589497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.589485    2551 server.go:1268] "Started kubelet"
Sep 15 21:08:21.589606 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.589560    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 21:08:21.589645 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.589630    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 21:08:21.589689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.589667    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 21:08:21.590511 ip-10-0-141-188 systemd[1]: Started Kubernetes Kubelet.
Sep 15 21:08:21.590971 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.590953    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 21:08:21.592437 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.592417    2551 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 21:08:21.594293 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.594272    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 21:08:21.597624 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.597605    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 21:08:21.597720 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.597624    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 21:08:21.597802 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.597785    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:21.598323 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598295    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 21:08:21.598604 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598582    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 21:08:21.598604 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598606    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 21:08:21.598727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598700    2551 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 21:08:21.598727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598710    2551 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 21:08:21.598820 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598791    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4t7vc"
Sep 15 21:08:21.599005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.598989    2551 factory.go:55] Registering systemd factory
Sep 15 21:08:21.599005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599016    2551 factory.go:223] Registration of the systemd container factory successfully
Sep 15 21:08:21.599285 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599271    2551 factory.go:153] Registering CRI-O factory
Sep 15 21:08:21.599375 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599289    2551 factory.go:223] Registration of the crio container factory successfully
Sep 15 21:08:21.599375 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599340    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 21:08:21.599487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599417    2551 factory.go:103] Registering Raw factory
Sep 15 21:08:21.599487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599434    2551 manager.go:1196] Started watching for new ooms in manager
Sep 15 21:08:21.599843 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.599824    2551 manager.go:349] Starting recovery of all containers
Sep 15 21:08:21.606525 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.606489    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 21:08:21.606747 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.606720    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 21:08:21.610753 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.606491    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-188.ec2.internal.18d59aae967dd53e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-188.ec2.internal,UID:ip-10-0-141-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-188.ec2.internal,},FirstTimestamp:2026-09-15 21:08:21.58945619 +0000 UTC m=+0.391325123,LastTimestamp:2026-09-15 21:08:21.58945619 +0000 UTC m=+0.391325123,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-188.ec2.internal,}"
Sep 15 21:08:21.616294 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.616277    2551 manager.go:354] Recovery completed
Sep 15 21:08:21.620690 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.620674    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.622718 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.622697    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.622790 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.622735    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.622790 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.622746    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.623309 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.623289    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 21:08:21.623309 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.623301    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 21:08:21.623386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.623320    2551 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:08:21.623386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.623368    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.624249 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.624232    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.624303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.624267    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.624303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.624292    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.624735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.624724    2551 policy_none.go:49] "None policy: Start"
Sep 15 21:08:21.624766 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.624740    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 21:08:21.624766 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.624750    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 21:08:21.625929 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.625918    2551 policy_none.go:47] "Start"
Sep 15 21:08:21.656917 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.656900    2551 manager.go:337] "Starting Device Plugin manager"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.656968    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.656982    2551 server.go:87] "Starting device plugin registration server"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.657289    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.657302    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.657404    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.657479    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.657485    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.658030    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 21:08:21.671700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.658078    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:21.757939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.757815    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.758998 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.758976    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.759112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.759011    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.759112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.759023    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.759112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.759048    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.760400 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.760379    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 21:08:21.761804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.761788    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 21:08:21.761883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.761838    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 21:08:21.761883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.761863    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 21:08:21.761944 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.761907    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 21:08:21.764135 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.764108    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:21.765444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.765429    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.765510 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.765450    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-188.ec2.internal\": node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:21.814843 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.814817    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:21.862045 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.862013    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"]
Sep 15 21:08:21.862180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.862128    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.865575 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.865557    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.865646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.865597    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.865646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.865618    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.867374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.867361    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.867500 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.867483    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.867554 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.867520    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.869161 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.869139    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.869300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.869168    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.869300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.869183    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.869300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.869212    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.869300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.869235    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.869300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.869248    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.870717 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.870662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.870717 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.870692    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:08:21.871555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.871542    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:08:21.871643 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.871566    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:08:21.871643 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.871576    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:08:21.900612 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.900588    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e305c17c251b0091afdabfaa1e8fa71a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal\" (UID: \"e305c17c251b0091afdabfaa1e8fa71a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.900735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.900626    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e305c17c251b0091afdabfaa1e8fa71a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal\" (UID: \"e305c17c251b0091afdabfaa1e8fa71a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.900735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:21.900686    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dab564a33b33063370e62af110a6955-config\") pod \"kube-apiserver-proxy-ip-10-0-141-188.ec2.internal\" (UID: \"7dab564a33b33063370e62af110a6955\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.902462 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.902447    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-188.ec2.internal\" not found" node="ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.906567 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.906553    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-188.ec2.internal\" not found" node="ip-10-0-141-188.ec2.internal"
Sep 15 21:08:21.915611 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:21.915596    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.001272 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.001243    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e305c17c251b0091afdabfaa1e8fa71a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal\" (UID: \"e305c17c251b0091afdabfaa1e8fa71a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.001383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.001277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e305c17c251b0091afdabfaa1e8fa71a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal\" (UID: \"e305c17c251b0091afdabfaa1e8fa71a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.001383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.001297    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dab564a33b33063370e62af110a6955-config\") pod \"kube-apiserver-proxy-ip-10-0-141-188.ec2.internal\" (UID: \"7dab564a33b33063370e62af110a6955\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.001383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.001336    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e305c17c251b0091afdabfaa1e8fa71a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal\" (UID: \"e305c17c251b0091afdabfaa1e8fa71a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.001383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.001339    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dab564a33b33063370e62af110a6955-config\") pod \"kube-apiserver-proxy-ip-10-0-141-188.ec2.internal\" (UID: \"7dab564a33b33063370e62af110a6955\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.001497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.001388    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e305c17c251b0091afdabfaa1e8fa71a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal\" (UID: \"e305c17c251b0091afdabfaa1e8fa71a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.016298 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.016246    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.117058 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.117020    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.205679 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.205637    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.209684 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.209667    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:22.217334 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.217309    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.317907 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.317822    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.418365 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.418334    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.518872 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.518846    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.529018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.528995    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 21:08:22.529168 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.529152    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:08:22.603343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.603264    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:03:22 +0000 UTC" deadline="2028-05-29 20:07:20.847522974 +0000 UTC"
Sep 15 21:08:22.603343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.603301    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14926h58m58.244225141s"
Sep 15 21:08:22.619268 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.619229    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.719858 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.719818    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.723041 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.723021    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:22.820051 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.819900    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:22.822220 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:22.821821    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dab564a33b33063370e62af110a6955.slice/crio-e1f9b3e4b4e9412e9d100e0c4e5688a333c880488e4b9e7bf375ee46138a95d6 WatchSource:0}: Error finding container e1f9b3e4b4e9412e9d100e0c4e5688a333c880488e4b9e7bf375ee46138a95d6: Status 404 returned error can't find the container with id e1f9b3e4b4e9412e9d100e0c4e5688a333c880488e4b9e7bf375ee46138a95d6
Sep 15 21:08:22.822220 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:22.822069    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode305c17c251b0091afdabfaa1e8fa71a.slice/crio-7b76c819cc4388d8b57b0ca882000845721fc53786aa6d186a406858c470bb2a WatchSource:0}: Error finding container 7b76c819cc4388d8b57b0ca882000845721fc53786aa6d186a406858c470bb2a: Status 404 returned error can't find the container with id 7b76c819cc4388d8b57b0ca882000845721fc53786aa6d186a406858c470bb2a
Sep 15 21:08:22.826691 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.826674    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:08:22.893272 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:22.893160    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:22.920628 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:22.920591    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:23.021105 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.021073    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-188.ec2.internal\" not found"
Sep 15 21:08:23.075588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.075556    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:23.099284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.099258    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:23.110830 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.110808    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:08:23.112038 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.112021    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"
Sep 15 21:08:23.120735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.120712    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:08:23.573458 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.573420    2551 apiserver.go:52] "Watching apiserver"
Sep 15 21:08:23.579930 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.579888    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 21:08:23.580495 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.580472    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-r9f4b","openshift-multus/multus-j45lb","openshift-multus/network-metrics-daemon-kwhrg","openshift-network-operator/iptables-alerter-2lnqq","kube-system/konnectivity-agent-nzr6t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz","openshift-cluster-node-tuning-operator/tuned-jrwrb","openshift-dns/node-resolver-xjwpf","openshift-image-registry/node-ca-p2hgv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal","openshift-network-diagnostics/network-check-target-t89pr","openshift-ovn-kubernetes/ovnkube-node-9qnx2","kube-system/global-pull-secret-syncer-t22b7","kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal"]
Sep 15 21:08:23.582651 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.582625    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.584467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.584445    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.584589 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.584534    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:23.585144 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.585124    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.585257 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.585124    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.585257 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.585181    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vksqf\""
Sep 15 21:08:23.589118 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.588982    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:23.589118 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.589076    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:23.590335 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.589711    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.591183 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.591163    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.592235 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.592218    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7259v\""
Sep 15 21:08:23.592468 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.592453    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 21:08:23.592618 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.592601    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.592758 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.592741    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.592881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.592861    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.593557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.593537    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hd2v\""
Sep 15 21:08:23.593657 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.593569    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 21:08:23.593783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.593766    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 21:08:23.594663 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.594644    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.595010 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.594980    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.595419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.595395    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 21:08:23.595519 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.595488    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.595598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.595579    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wm2rn\""
Sep 15 21:08:23.596394 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.596375    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.597366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.597008    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hv2xw\""
Sep 15 21:08:23.597366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.597161    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 21:08:23.597366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.597179    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.597654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.597455    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.598488 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.598464    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.598488 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.598481    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 21:08:23.599156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.599139    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 21:08:23.599419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.599402    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 21:08:23.599419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.599413    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.599699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.599684    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 21:08:23.599770 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.599703    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2tzc4\""
Sep 15 21:08:23.600096 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.600080    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.600369 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.600348    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.600547 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.600529    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 21:08:23.600621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.600556    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bs4hx\""
Sep 15 21:08:23.603038 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.603015    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:23.603137 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.603105    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:23.604820 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.604795    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sg7ng\""
Sep 15 21:08:23.604820 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.604811    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.605165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.605127    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.607446 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.607424    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.609879 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.609860    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 21:08:23.610095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610074    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 21:08:23.610396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610371    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-k8s-cni-cncf-io\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.610512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysctl-d\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.610512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610436    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-run\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.610512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610483    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:08:23.610670 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610659    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zwml2\""
Sep 15 21:08:23.610717 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610696    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 21:08:23.610914 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610488    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-tuned\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.610914 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610864    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 21:08:23.610914 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610888    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7smk\" (UniqueName: \"kubernetes.io/projected/05ad11b2-75b4-4a23-91cd-33be656bd8f6-kube-api-access-n7smk\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.611105 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.610936    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-iptables-alerter-script\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.611105 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611004    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-modprobe-d\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.611105 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-dbus\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.611245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611133    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.611245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611152    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 21:08:23.611245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.611245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611221    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02ddff51-d8f4-4565-9e73-2d6796a09b90-agent-certs\") pod \"konnectivity-agent-nzr6t\" (UID: \"02ddff51-d8f4-4565-9e73-2d6796a09b90\") " pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.611444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611268    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-host-slash\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.611444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611294    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-registration-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.611444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611318    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.611444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611342    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-serviceca\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.611444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611402    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5c9s\" (UniqueName: \"kubernetes.io/projected/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-kube-api-access-b5c9s\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.611444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611438    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60e6c8dc-a917-46e1-9842-fbb017cf2517-cni-binary-copy\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cnibin\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611536    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-cni-bin\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611589    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-hostroot\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611650    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-system-cni-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611664    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611692    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-multus-certs\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysctl-conf\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611741    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-systemd\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.611771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611761    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611791    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzf6m\" (UniqueName: \"kubernetes.io/projected/6c4ca789-e2fc-4948-8395-3a2fbbf49059-kube-api-access-vzf6m\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611806    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-netns\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611833    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-lib-modules\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611856    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgp7m\" (UniqueName: \"kubernetes.io/projected/4665f784-4a2c-4e3f-887d-0e0fc600191f-kube-api-access-zgp7m\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611878    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysconfig\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611901    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4665f784-4a2c-4e3f-887d-0e0fc600191f-tmp\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611965    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.611999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-socket-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612072    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05ad11b2-75b4-4a23-91cd-33be656bd8f6-hosts-file\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612175    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-socket-dir-parent\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612242    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612268    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7zx\" (UniqueName: \"kubernetes.io/projected/4f11c43a-261c-47bf-9fb4-842609d11c3e-kube-api-access-dt7zx\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612293    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-kubelet\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612319    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-cni-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612342    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-kubelet-config\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612368    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ad11b2-75b4-4a23-91cd-33be656bd8f6-tmp-dir\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612384    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7zf\" (UniqueName: \"kubernetes.io/projected/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-kube-api-access-bq7zf\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612398    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-cnibin\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612412    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-sys-fs\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612471    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-host\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-etc-kubernetes\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612516    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612530    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-kubernetes\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4xr\" (UniqueName: \"kubernetes.io/projected/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-kube-api-access-wl4xr\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612583    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-cni-multus\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.612842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612610    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-device-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612639    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-daemon-config\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612666    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rtzn\" (UniqueName: \"kubernetes.io/projected/60e6c8dc-a917-46e1-9842-fbb017cf2517-kube-api-access-5rtzn\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-var-lib-kubelet\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612763    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02ddff51-d8f4-4565-9e73-2d6796a09b90-konnectivity-ca\") pod \"konnectivity-agent-nzr6t\" (UID: \"02ddff51-d8f4-4565-9e73-2d6796a09b90\") " pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-system-cni-dir\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612829    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-conf-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612850    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-host\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-sys\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.613558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.612905    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.699101 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.699073    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 21:08:23.713171 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713137    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.713171 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713182    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-sys-fs\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713210    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-host\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-run-netns\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713284    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-ovn\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713313    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj595\" (UniqueName: \"kubernetes.io/projected/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-kube-api-access-bj595\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-etc-kubernetes\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:23.713416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-kubernetes\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4xr\" (UniqueName: \"kubernetes.io/projected/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-kube-api-access-wl4xr\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713461    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-cni-multus\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713487    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-device-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713519    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-daemon-config\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rtzn\" (UniqueName: \"kubernetes.io/projected/60e6c8dc-a917-46e1-9842-fbb017cf2517-kube-api-access-5rtzn\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-var-lib-kubelet\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713568    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713598    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02ddff51-d8f4-4565-9e73-2d6796a09b90-konnectivity-ca\") pod \"konnectivity-agent-nzr6t\" (UID: \"02ddff51-d8f4-4565-9e73-2d6796a09b90\") " pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-system-cni-dir\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713646    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-sys-fs\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713648    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-conf-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713701    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-etc-kubernetes\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713706    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-conf-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.713755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.713757    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-host\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714050    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-device-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714069    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-kubernetes\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714088    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-host\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714105    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-sys\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714125    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714145    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-cni-bin\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714162    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-k8s-cni-cncf-io\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714178    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysctl-d\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714234    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-run\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-tuned\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714296    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7smk\" (UniqueName: \"kubernetes.io/projected/05ad11b2-75b4-4a23-91cd-33be656bd8f6-kube-api-access-n7smk\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-slash\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714347    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714373    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714401    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-iptables-alerter-script\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714427    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-modprobe-d\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.714435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714442    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-dbus\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714459    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714485    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714507    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-etc-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714665    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovn-node-metrics-cert\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02ddff51-d8f4-4565-9e73-2d6796a09b90-agent-certs\") pod \"konnectivity-agent-nzr6t\" (UID: \"02ddff51-d8f4-4565-9e73-2d6796a09b90\") " pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714702    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-host-slash\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714717    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-daemon-config\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714720    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-registration-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-registration-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714787    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-serviceca\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714845    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5c9s\" (UniqueName: \"kubernetes.io/projected/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-kube-api-access-b5c9s\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714864    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysctl-d\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714874    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60e6c8dc-a917-46e1-9842-fbb017cf2517-cni-binary-copy\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714908    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cnibin\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714943    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-cni-bin\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.715312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714969    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-hostroot\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.714996    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-system-cni-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715037    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-multus-certs\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715064    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysctl-conf\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715082    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-cni-multus\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-systemd\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715124    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-host\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715155    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzf6m\" (UniqueName: \"kubernetes.io/projected/6c4ca789-e2fc-4948-8395-3a2fbbf49059-kube-api-access-vzf6m\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-k8s-cni-cncf-io\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-sys\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715381    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-run\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715966    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.715990    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-var-lib-kubelet\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716035    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-system-cni-dir\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.716147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716134    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716425    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysctl-conf\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716500    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716504    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02ddff51-d8f4-4565-9e73-2d6796a09b90-konnectivity-ca\") pod \"konnectivity-agent-nzr6t\" (UID: \"02ddff51-d8f4-4565-9e73-2d6796a09b90\") " pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716605    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-modprobe-d\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716687    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-dbus\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-system-cni-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-serviceca\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716735    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-netns\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716814    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-systemd\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716841    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-hostroot\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716813    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-lib-modules\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.716942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716930    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgp7m\" (UniqueName: \"kubernetes.io/projected/4665f784-4a2c-4e3f-887d-0e0fc600191f-kube-api-access-zgp7m\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.717005    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717069    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-var-lib-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717105    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717146    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovnkube-script-lib\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.716777    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cnibin\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.717228    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:08:24.217175517 +0000 UTC m=+3.019044441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysconfig\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4665f784-4a2c-4e3f-887d-0e0fc600191f-tmp\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717358    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717386    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-kubelet\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717425    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-systemd-units\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717442    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717462    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-env-overrides\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.717588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717497    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-socket-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717641    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05ad11b2-75b4-4a23-91cd-33be656bd8f6-hosts-file\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717673    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-node-log\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717703    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-log-socket\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717708    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-iptables-alerter-script\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-socket-dir-parent\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717745    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60e6c8dc-a917-46e1-9842-fbb017cf2517-cni-binary-copy\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717793    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-host-slash\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717793    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7zx\" (UniqueName: \"kubernetes.io/projected/4f11c43a-261c-47bf-9fb4-842609d11c3e-kube-api-access-dt7zx\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717790    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c4ca789-e2fc-4948-8395-3a2fbbf49059-socket-dir\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717847    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-multus-certs\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717899    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-sysconfig\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4665f784-4a2c-4e3f-887d-0e0fc600191f-lib-modules\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717928    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-cni-bin\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717984    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-systemd\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718013    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-cni-netd\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.718311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718056    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-socket-dir-parent\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718062    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovnkube-config\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.717985    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05ad11b2-75b4-4a23-91cd-33be656bd8f6-hosts-file\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718158    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-kubelet\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718229    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-cni-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718284    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-kubelet-config\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ad11b2-75b4-4a23-91cd-33be656bd8f6-tmp-dir\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718444    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7zf\" (UniqueName: \"kubernetes.io/projected/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-kube-api-access-bq7zf\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718475    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-cnibin\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.718487    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718586    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-cnibin\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.718595    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:08:24.21857126 +0000 UTC m=+3.020440183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718691    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-multus-cni-dir\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718694    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-run-netns\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718795    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-kubelet-config\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.718914    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60e6c8dc-a917-46e1-9842-fbb017cf2517-host-var-lib-kubelet\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.719856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.719187    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ad11b2-75b4-4a23-91cd-33be656bd8f6-tmp-dir\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.720723 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.720141    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:23.720723 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.720176    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:23.720723 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.720188    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:23.720723 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:23.720261    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:24.220243088 +0000 UTC m=+3.022112012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:23.721484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.721455    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4665f784-4a2c-4e3f-887d-0e0fc600191f-tmp\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.721757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.721625    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4665f784-4a2c-4e3f-887d-0e0fc600191f-etc-tuned\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.722061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.721843    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02ddff51-d8f4-4565-9e73-2d6796a09b90-agent-certs\") pod \"konnectivity-agent-nzr6t\" (UID: \"02ddff51-d8f4-4565-9e73-2d6796a09b90\") " pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.723898 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.723846    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rtzn\" (UniqueName: \"kubernetes.io/projected/60e6c8dc-a917-46e1-9842-fbb017cf2517-kube-api-access-5rtzn\") pod \"multus-j45lb\" (UID: \"60e6c8dc-a917-46e1-9842-fbb017cf2517\") " pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.724408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.724386    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4xr\" (UniqueName: \"kubernetes.io/projected/36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29-kube-api-access-wl4xr\") pod \"multus-additional-cni-plugins-r9f4b\" (UID: \"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29\") " pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.729780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.729755    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7smk\" (UniqueName: \"kubernetes.io/projected/05ad11b2-75b4-4a23-91cd-33be656bd8f6-kube-api-access-n7smk\") pod \"node-resolver-xjwpf\" (UID: \"05ad11b2-75b4-4a23-91cd-33be656bd8f6\") " pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.730541 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.730514    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5c9s\" (UniqueName: \"kubernetes.io/projected/7b9309ce-ed18-4f88-8b8e-28074c47e3f2-kube-api-access-b5c9s\") pod \"node-ca-p2hgv\" (UID: \"7b9309ce-ed18-4f88-8b8e-28074c47e3f2\") " pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.741546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.741519    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzf6m\" (UniqueName: \"kubernetes.io/projected/6c4ca789-e2fc-4948-8395-3a2fbbf49059-kube-api-access-vzf6m\") pod \"aws-ebs-csi-driver-node-lmmkz\" (UID: \"6c4ca789-e2fc-4948-8395-3a2fbbf49059\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.744657 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.744352    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgp7m\" (UniqueName: \"kubernetes.io/projected/4665f784-4a2c-4e3f-887d-0e0fc600191f-kube-api-access-zgp7m\") pod \"tuned-jrwrb\" (UID: \"4665f784-4a2c-4e3f-887d-0e0fc600191f\") " pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.745315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.745291    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7zf\" (UniqueName: \"kubernetes.io/projected/7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1-kube-api-access-bq7zf\") pod \"iptables-alerter-2lnqq\" (UID: \"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1\") " pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.745802 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.745778    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7zx\" (UniqueName: \"kubernetes.io/projected/4f11c43a-261c-47bf-9fb4-842609d11c3e-kube-api-access-dt7zx\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:23.746995 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.746977    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mpptw"
Sep 15 21:08:23.754035 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.754003    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mpptw"
Sep 15 21:08:23.761914 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.761886    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:08:23.767668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.767610    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal" event={"ID":"e305c17c251b0091afdabfaa1e8fa71a","Type":"ContainerStarted","Data":"7b76c819cc4388d8b57b0ca882000845721fc53786aa6d186a406858c470bb2a"}
Sep 15 21:08:23.769137 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.769109    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal" event={"ID":"7dab564a33b33063370e62af110a6955","Type":"ContainerStarted","Data":"e1f9b3e4b4e9412e9d100e0c4e5688a333c880488e4b9e7bf375ee46138a95d6"}
Sep 15 21:08:23.819124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-kubelet\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819136    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-systemd-units\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819165    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819190    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-env-overrides\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819240    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-node-log\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819266    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-log-socket\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819299    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-systemd-units\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819301    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-kubelet\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-systemd\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819357    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-node-log\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819371    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-systemd\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819377    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-cni-netd\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819404    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-log-socket\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819408    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovnkube-config\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819437    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819442    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-run-netns\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819467    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-ovn\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-cni-netd\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819511    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj595\" (UniqueName: \"kubernetes.io/projected/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-kube-api-access-bj595\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819545    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-run-netns\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819569    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-cni-bin\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819601    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-slash\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819626    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-etc-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovn-node-metrics-cert\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819766    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-var-lib-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.819893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819790    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovnkube-script-lib\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-env-overrides\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.819959    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820022    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-run-ovn\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820121    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovnkube-config\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820187    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-etc-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820230    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-cni-bin\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820345    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-host-slash\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820349    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovnkube-script-lib\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.820454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.820398    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-var-lib-openvswitch\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.825366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.825267    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-ovn-node-metrics-cert\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.832443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.832417    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj595\" (UniqueName: \"kubernetes.io/projected/4dd46b1d-fc21-4d24-a70a-befb4b04c3b1-kube-api-access-bj595\") pod \"ovnkube-node-9qnx2\" (UID: \"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:23.897196 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.896910    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jrwrb"
Sep 15 21:08:23.911204 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.911175    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2lnqq"
Sep 15 21:08:23.920317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.920289    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:23.926927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.926743    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz"
Sep 15 21:08:23.934838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.934808    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p2hgv"
Sep 15 21:08:23.945081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.945053    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r9f4b"
Sep 15 21:08:23.952184 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.952161    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j45lb"
Sep 15 21:08:23.961393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.961367    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjwpf"
Sep 15 21:08:23.966816 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:23.966615    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:24.223621    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:24.223693    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:24.223754    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223764    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223836    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:08:25.223819379 +0000 UTC m=+4.025688301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223856    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223873    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223895    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223909    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223922    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:08:25.223902046 +0000 UTC m=+4.025770993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:24.224001 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:24.223963    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:25.223948003 +0000 UTC m=+4.025816925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:24.757211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:24.757154    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:03:24 +0000 UTC" deadline="2028-05-08 07:42:04.20781526 +0000 UTC"
Sep 15 21:08:24.757211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:24.757216    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14410h33m39.450604732s"
Sep 15 21:08:24.833734 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.833706    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f5e3ee_6ca4_4799_81d8_bbd4e1cdfd29.slice/crio-76c0120b8b746e2a23b4154261d105fa49c4fc7c3de8d602e9a93f8cb531caf4 WatchSource:0}: Error finding container 76c0120b8b746e2a23b4154261d105fa49c4fc7c3de8d602e9a93f8cb531caf4: Status 404 returned error can't find the container with id 76c0120b8b746e2a23b4154261d105fa49c4fc7c3de8d602e9a93f8cb531caf4
Sep 15 21:08:24.834903 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.834874    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4665f784_4a2c_4e3f_887d_0e0fc600191f.slice/crio-37553ce137b2f82e5d582ea93ddfdf45e8dfcac098da9caec5d5a6b47e22ff9c WatchSource:0}: Error finding container 37553ce137b2f82e5d582ea93ddfdf45e8dfcac098da9caec5d5a6b47e22ff9c: Status 404 returned error can't find the container with id 37553ce137b2f82e5d582ea93ddfdf45e8dfcac098da9caec5d5a6b47e22ff9c
Sep 15 21:08:24.836489 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.836448    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7dfae9_28f0_4ddf_8351_18ea6ebbfaa1.slice/crio-5a5e5584052010f16359bb318410860d45cf45f4afe053eaa9bed023ee13ba3f WatchSource:0}: Error finding container 5a5e5584052010f16359bb318410860d45cf45f4afe053eaa9bed023ee13ba3f: Status 404 returned error can't find the container with id 5a5e5584052010f16359bb318410860d45cf45f4afe053eaa9bed023ee13ba3f
Sep 15 21:08:24.838220 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.838191    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9309ce_ed18_4f88_8b8e_28074c47e3f2.slice/crio-33002e50c620527fe1a1b9afc3e832817700701d87b2e13fd8b00133106069d1 WatchSource:0}: Error finding container 33002e50c620527fe1a1b9afc3e832817700701d87b2e13fd8b00133106069d1: Status 404 returned error can't find the container with id 33002e50c620527fe1a1b9afc3e832817700701d87b2e13fd8b00133106069d1
Sep 15 21:08:24.842308 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.842272    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e6c8dc_a917_46e1_9842_fbb017cf2517.slice/crio-c6a822bdd840dc0e4395e3a8217325e7ccdbc206da5b11e0ee8f86952f47c470 WatchSource:0}: Error finding container c6a822bdd840dc0e4395e3a8217325e7ccdbc206da5b11e0ee8f86952f47c470: Status 404 returned error can't find the container with id c6a822bdd840dc0e4395e3a8217325e7ccdbc206da5b11e0ee8f86952f47c470
Sep 15 21:08:24.843155 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.843133    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd46b1d_fc21_4d24_a70a_befb4b04c3b1.slice/crio-2c064289a0fdfd2765afd8d93430c218d022749a02c140a2bac623e3f955c027 WatchSource:0}: Error finding container 2c064289a0fdfd2765afd8d93430c218d022749a02c140a2bac623e3f955c027: Status 404 returned error can't find the container with id 2c064289a0fdfd2765afd8d93430c218d022749a02c140a2bac623e3f955c027
Sep 15 21:08:24.848152 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.847579    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4ca789_e2fc_4948_8395_3a2fbbf49059.slice/crio-55884b0da9142d472d48cc395e1e8f2c2ac92fb9093e8bd8571680de375429ad WatchSource:0}: Error finding container 55884b0da9142d472d48cc395e1e8f2c2ac92fb9093e8bd8571680de375429ad: Status 404 returned error can't find the container with id 55884b0da9142d472d48cc395e1e8f2c2ac92fb9093e8bd8571680de375429ad
Sep 15 21:08:24.851183 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:24.851157    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ddff51_d8f4_4565_9e73_2d6796a09b90.slice/crio-97b3af513350f0470a938a35e3db34b9adfb566253da1ee18ce1d437ab491105 WatchSource:0}: Error finding container 97b3af513350f0470a938a35e3db34b9adfb566253da1ee18ce1d437ab491105: Status 404 returned error can't find the container with id 97b3af513350f0470a938a35e3db34b9adfb566253da1ee18ce1d437ab491105
Sep 15 21:08:25.231943 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.231897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:25.232144 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.231991    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:25.232144 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.232020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:25.232144 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232112    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:25.232144 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232109    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:25.232144 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232133    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:25.232144 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232143    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:25.232417 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232162    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:25.232417 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232168    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:08:27.232154868 +0000 UTC m=+6.034023794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:25.232417 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232189    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:08:27.23217561 +0000 UTC m=+6.034044537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:25.232417 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.232245    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:27.232206223 +0000 UTC m=+6.034075165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:25.757481 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.757379    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:03:24 +0000 UTC" deadline="2028-02-24 17:14:57.6760658 +0000 UTC"
Sep 15 21:08:25.757481 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.757428    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12644h6m31.918642701s"
Sep 15 21:08:25.765405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.765377    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:25.765550 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.765510    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:25.765972 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.765950    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:25.766099 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.766075    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:25.766164 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.766151    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:25.766241 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:25.766223    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:25.791436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.791402    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerStarted","Data":"76c0120b8b746e2a23b4154261d105fa49c4fc7c3de8d602e9a93f8cb531caf4"}
Sep 15 21:08:25.805548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.805512    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nzr6t" event={"ID":"02ddff51-d8f4-4565-9e73-2d6796a09b90","Type":"ContainerStarted","Data":"97b3af513350f0470a938a35e3db34b9adfb566253da1ee18ce1d437ab491105"}
Sep 15 21:08:25.810362 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.810324    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjwpf" event={"ID":"05ad11b2-75b4-4a23-91cd-33be656bd8f6","Type":"ContainerStarted","Data":"276c8b881401e94c9c2f9fe5264ba081bbfc00b7e1e326e1e6eef5cad659a5f3"}
Sep 15 21:08:25.817686 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.817645    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"2c064289a0fdfd2765afd8d93430c218d022749a02c140a2bac623e3f955c027"}
Sep 15 21:08:25.821085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.821047    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j45lb" event={"ID":"60e6c8dc-a917-46e1-9842-fbb017cf2517","Type":"ContainerStarted","Data":"c6a822bdd840dc0e4395e3a8217325e7ccdbc206da5b11e0ee8f86952f47c470"}
Sep 15 21:08:25.825534 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.825502    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lnqq" event={"ID":"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1","Type":"ContainerStarted","Data":"5a5e5584052010f16359bb318410860d45cf45f4afe053eaa9bed023ee13ba3f"}
Sep 15 21:08:25.836184 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.836147    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal" event={"ID":"7dab564a33b33063370e62af110a6955","Type":"ContainerStarted","Data":"4edece2098f987d0f3f1bbcff40b7a381875e86eca79ce1974abf9d1aba93ce3"}
Sep 15 21:08:25.841946 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.841439    2551 generic.go:356] "Generic (PLEG): container finished" podID="e305c17c251b0091afdabfaa1e8fa71a" containerID="24e1ad737d0e26801d8937ef12dbb4576b239cf755fce8f64bf60d71f4e6d776" exitCode=0
Sep 15 21:08:25.841946 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.841534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal" event={"ID":"e305c17c251b0091afdabfaa1e8fa71a","Type":"ContainerDied","Data":"24e1ad737d0e26801d8937ef12dbb4576b239cf755fce8f64bf60d71f4e6d776"}
Sep 15 21:08:25.843753 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.843670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz" event={"ID":"6c4ca789-e2fc-4948-8395-3a2fbbf49059","Type":"ContainerStarted","Data":"55884b0da9142d472d48cc395e1e8f2c2ac92fb9093e8bd8571680de375429ad"}
Sep 15 21:08:25.851488 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.851453    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p2hgv" event={"ID":"7b9309ce-ed18-4f88-8b8e-28074c47e3f2","Type":"ContainerStarted","Data":"33002e50c620527fe1a1b9afc3e832817700701d87b2e13fd8b00133106069d1"}
Sep 15 21:08:25.868579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.868543    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jrwrb" event={"ID":"4665f784-4a2c-4e3f-887d-0e0fc600191f","Type":"ContainerStarted","Data":"37553ce137b2f82e5d582ea93ddfdf45e8dfcac098da9caec5d5a6b47e22ff9c"}
Sep 15 21:08:25.878527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:25.878465    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-188.ec2.internal" podStartSLOduration=2.878443418 podStartE2EDuration="2.878443418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:25.858076008 +0000 UTC m=+4.659944955" watchObservedRunningTime="2026-09-15 21:08:25.878443418 +0000 UTC m=+4.680312366"
Sep 15 21:08:26.891165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:26.890877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal" event={"ID":"e305c17c251b0091afdabfaa1e8fa71a","Type":"ContainerStarted","Data":"0422a100e8262dd8c05f1a8b674c506eae35af46e8d3b23184551679c3703fbb"}
Sep 15 21:08:27.251786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:27.251690    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:27.251786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:27.251763    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:27.252031 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:27.251804    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:27.252031 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.251959    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:27.252031 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.251978    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:27.252031 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.251991    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:27.252249 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.252067    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:31.252037716 +0000 UTC m=+10.053906642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:27.252608 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.252530    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:27.252608 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.252595    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:08:31.252577177 +0000 UTC m=+10.054446103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:27.252781 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.252652    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:27.252781 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.252687    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:08:31.252675382 +0000 UTC m=+10.054544312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:27.765017 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:27.763421    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:27.765017 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:27.763422    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:27.765017 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.763573    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:27.765017 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:27.763618    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:27.765017 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.763710    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:27.765017 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:27.763800    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:29.763387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:29.763339    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:29.763387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:29.763366    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:29.763956 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:29.763492    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:29.763956 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:29.763593    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:29.763956 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:29.763662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:29.763956 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:29.763758    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:31.286341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:31.286304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:31.286366    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:31.286437    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286491    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286553    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286566    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:08:39.286550903 +0000 UTC m=+18.088419825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286610    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:08:39.286590873 +0000 UTC m=+18.088459798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286695    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286710    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286724    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:31.286798 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.286761    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:39.286748928 +0000 UTC m=+18.088617873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:31.764071 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:31.763981    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:31.764273 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.764080    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:31.764273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:31.764153    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:31.764415 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.764308    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:31.764415 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:31.764410    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:31.764527 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:31.764507    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:33.765813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:33.765089    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:33.765813 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:33.765234    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:33.765813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:33.765304    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:33.765813 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:33.765388    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:33.765813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:33.765426    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:33.765813 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:33.765485    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:35.762487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:35.762445    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:35.762487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:35.762468    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:35.762962 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:35.762571    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:35.762962 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:35.762583    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:35.762962 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:35.762682    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:35.762962 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:35.762758    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:37.762933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:37.762893    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:37.762933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:37.762912    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:37.763471 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:37.763017    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:37.763471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:37.763056    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:37.763471 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:37.763163    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:37.763471 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:37.763305    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:39.352147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:39.352103    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:39.352168    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:39.352235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352274    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352356    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:08:55.352336109 +0000 UTC m=+34.154205042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352404    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352426    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352452    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352467    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352456    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:08:55.352439008 +0000 UTC m=+34.154307933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:39.352700 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.352533    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:55.352520116 +0000 UTC m=+34.154389036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:39.762681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:39.762163    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:39.762681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:39.762171    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:39.762681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:39.762312    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:39.762681 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.762308    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:39.762681 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.762412    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:39.762681 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:39.762495    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:41.763923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:41.763883    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:41.764410 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:41.764061    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:41.764578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:41.764558    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:41.764701 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:41.764681    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:41.764766 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:41.764752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:41.764902 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:41.764885    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:43.762247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.762069    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:43.762247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.762076    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:43.762247 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:43.762206    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:43.762247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.762221    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:43.762765 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:43.762321    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:43.762765 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:43.762413    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:43.931326 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.930629    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p2hgv" event={"ID":"7b9309ce-ed18-4f88-8b8e-28074c47e3f2","Type":"ContainerStarted","Data":"9c17b958bb96076e4b4aab9683c02fb82fb1f5c1a95cb1f1dca91a799216fa84"}
Sep 15 21:08:43.933951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.933919    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jrwrb" event={"ID":"4665f784-4a2c-4e3f-887d-0e0fc600191f","Type":"ContainerStarted","Data":"988769047c6cea64670f1ec368ccf5ef8df13439047c46169d063c5772ac4874"}
Sep 15 21:08:43.948926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.948399    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p2hgv" podStartSLOduration=3.051241448 podStartE2EDuration="21.948373485s" totalImagesPullingTime="18.897132037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:43.947905572 +0000 UTC m=+22.749774523" watchObservedRunningTime="2026-09-15 21:08:43.948373485 +0000 UTC m=+22.750242429"
Sep 15 21:08:43.948926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.948507    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-188.ec2.internal" podStartSLOduration=20.948498 podStartE2EDuration="20.948498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:26.908302788 +0000 UTC m=+5.710171764" watchObservedRunningTime="2026-09-15 21:08:43.948498 +0000 UTC m=+22.750366946"
Sep 15 21:08:43.972412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:43.972358    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jrwrb" podStartSLOduration=3.061604688 podStartE2EDuration="21.972343585s" totalImagesPullingTime="18.910738897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:43.971932167 +0000 UTC m=+22.773801136" watchObservedRunningTime="2026-09-15 21:08:43.972343585 +0000 UTC m=+22.774212528"
Sep 15 21:08:44.939241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.938985    2551 generic.go:356] "Generic (PLEG): container finished" podID="36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29" containerID="b79e4bdfb7fb4ab79a319eda3a899ec8f3df09d4b411679740af6fb3ce947c29" exitCode=0
Sep 15 21:08:44.940383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.939074    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerDied","Data":"b79e4bdfb7fb4ab79a319eda3a899ec8f3df09d4b411679740af6fb3ce947c29"}
Sep 15 21:08:44.941300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.941266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nzr6t" event={"ID":"02ddff51-d8f4-4565-9e73-2d6796a09b90","Type":"ContainerStarted","Data":"249047a342fe306a795ef1b4f0ffab13e5b54beadf9b67d8cf74864ee397c06f"}
Sep 15 21:08:44.942954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.942930    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjwpf" event={"ID":"05ad11b2-75b4-4a23-91cd-33be656bd8f6","Type":"ContainerStarted","Data":"a12d586644a4316a34595bda937cc7f94408ca191d112d87763a30a88d2ccdb7"}
Sep 15 21:08:44.947242 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.947218    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"a6c7529640a75196b98c470bfdd911bd6063c2c8fa4b59b808118142b26429e8"}
Sep 15 21:08:44.947343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.947304    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"86347cd754f96da64486489d811694762af59b3790eb43df9d6f31da7bbb2cba"}
Sep 15 21:08:44.947405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.947363    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"48a9510a9dac7f9285b673eca317794d21b30437394b8077fcb40cdb29c4ab80"}
Sep 15 21:08:44.947450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.947414    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"d8a67e351ccff7b3946dbbe397f783e8ea0c62c52f551bbe027da1e9823b8a33"}
Sep 15 21:08:44.947450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.947446    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"9dc6c86d1deba0f709ca232b3c4e06c0f1c656ae515283337a062941e517cf39"}
Sep 15 21:08:44.947547 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.947485    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"7a315d855bb7f823d351afe8d732747a657fa495112fc6ef715376fbf3da9e95"}
Sep 15 21:08:44.948922 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.948892    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j45lb" event={"ID":"60e6c8dc-a917-46e1-9842-fbb017cf2517","Type":"ContainerStarted","Data":"aa7790bcf0c32edf3aafeb46058b6eebafce42479b0c1bb8d7a04155c89395b3"}
Sep 15 21:08:44.950382 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.950333    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz" event={"ID":"6c4ca789-e2fc-4948-8395-3a2fbbf49059","Type":"ContainerStarted","Data":"a4996cd87b5e9ec1b89c2c63e54213aa6c072ec7dd3fcadf9b43d0607366af92"}
Sep 15 21:08:44.978569 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:44.978516    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nzr6t" podStartSLOduration=4.095549828 podStartE2EDuration="22.978500427s" totalImagesPullingTime="18.882950599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:44.977817441 +0000 UTC m=+23.779686402" watchObservedRunningTime="2026-09-15 21:08:44.978500427 +0000 UTC m=+23.780369370"
Sep 15 21:08:45.009622 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.009577    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xjwpf" podStartSLOduration=4.109498785 podStartE2EDuration="23.009561269s" totalImagesPullingTime="18.900062484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:45.009452026 +0000 UTC m=+23.811320990" watchObservedRunningTime="2026-09-15 21:08:45.009561269 +0000 UTC m=+23.811430212"
Sep 15 21:08:45.010021 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.009990    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j45lb" podStartSLOduration=4.042468207 podStartE2EDuration="23.009980464s" totalImagesPullingTime="18.967512257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:44.993809145 +0000 UTC m=+23.795678111" watchObservedRunningTime="2026-09-15 21:08:45.009980464 +0000 UTC m=+23.811849406"
Sep 15 21:08:45.156684 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.156494    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:45.207600 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.207498    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:45.362030 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.361999    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 21:08:45.673645 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.673554    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:08:45.362017406Z","UUID":"d7c1b52b-be52-42ea-a1f1-64537e5f4786","Handler":null,"Name":"","Endpoint":""}
Sep 15 21:08:45.675228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.675186    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 21:08:45.675364 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.675244    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 21:08:45.762413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.762384    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:45.762595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.762428    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:45.762595 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:45.762498    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:45.762595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.762383    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:45.762595 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:45.762586    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:45.762770 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:45.762660    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:45.840061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.840019    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:45.841039 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.841015    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:45.957968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.957855    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz" event={"ID":"6c4ca789-e2fc-4948-8395-3a2fbbf49059","Type":"ContainerStarted","Data":"fc879732e31de9ea1cd0c3d3b4fbf76f08df7871564adc466ab9cd680a1fd280"}
Sep 15 21:08:45.960092 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.960057    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2lnqq" event={"ID":"7a7dfae9-28f0-4ddf-8351-18ea6ebbfaa1","Type":"ContainerStarted","Data":"9414bf69a8b5382e21a59a192febad431b92756b475aa9cd6bbcb7fbf1baf04f"}
Sep 15 21:08:45.960576 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.960484    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:45.960949 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.960932    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nzr6t"
Sep 15 21:08:45.974333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:45.974280    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2lnqq" podStartSLOduration=5.059516135 podStartE2EDuration="23.974258648s" totalImagesPullingTime="18.914742513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:45.973830372 +0000 UTC m=+24.775699340" watchObservedRunningTime="2026-09-15 21:08:45.974258648 +0000 UTC m=+24.776127594"
Sep 15 21:08:46.042768 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:46.042716    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:46.144586 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:46.144547    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:46.213601 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:46.213510    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:46.965428 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:46.965389    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz" event={"ID":"6c4ca789-e2fc-4948-8395-3a2fbbf49059","Type":"ContainerStarted","Data":"e797a4286558d25ad198333d2bfc5a651879767223d26dd7f6492619d21074a4"}
Sep 15 21:08:46.969508 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:46.969482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"9bacc42579cd00f0a2e82c26951a7f140a1868c1960b67ed115cc666f2ad69d0"}
Sep 15 21:08:46.984721 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:46.984661    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmmkz" podStartSLOduration=3.83832427 podStartE2EDuration="24.984637411s" totalImagesPullingTime="21.146313141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:46.982650084 +0000 UTC m=+25.784519048" watchObservedRunningTime="2026-09-15 21:08:46.984637411 +0000 UTC m=+25.786506359"
Sep 15 21:08:47.762696 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:47.762662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:47.762696 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:47.762705    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:47.762988 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:47.762662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:47.762988 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:47.762809    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:47.762988 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:47.762934    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:47.763142 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:47.763076    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:48.290666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:48.290625    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log" limit=1024
Sep 15 21:08:49.763141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.762877    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:49.764047 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:49.763149    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:49.764047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.762877    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:49.764047 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:49.763237    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:49.764047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.762877    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:49.764047 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:49.763289    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:49.980741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.980719    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" event={"ID":"4dd46b1d-fc21-4d24-a70a-befb4b04c3b1","Type":"ContainerStarted","Data":"f0d605d7204efa09abb8ca0efdebf23e954c2b8b93c110beaee7717fb77eea97"}
Sep 15 21:08:49.980985 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.980971    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:49.981043 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.981028    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:49.996401 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:49.996380    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:50.009993 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:50.009949    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2" podStartSLOduration=9.048202194 podStartE2EDuration="28.009932929s" totalImagesPullingTime="18.961730735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:08:50.009570033 +0000 UTC m=+28.811438977" watchObservedRunningTime="2026-09-15 21:08:50.009932929 +0000 UTC m=+28.811801873"
Sep 15 21:08:50.985581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:50.985535    2551 generic.go:356] "Generic (PLEG): container finished" podID="36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29" containerID="2737f105c192e2c016c33e1e6ca1f4ffdd74b6066817d8597510a1167799e205" exitCode=0
Sep 15 21:08:50.987509 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:50.987472    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerDied","Data":"2737f105c192e2c016c33e1e6ca1f4ffdd74b6066817d8597510a1167799e205"}
Sep 15 21:08:50.987681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:50.987660    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:51.012431 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.012405    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:08:51.596903 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.596716    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kwhrg"]
Sep 15 21:08:51.597116 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.597012    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:51.597170 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:51.597137    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:51.600589 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.600560    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t22b7"]
Sep 15 21:08:51.600740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.600689    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:51.600812 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:51.600790    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:51.601389 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.601353    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t89pr"]
Sep 15 21:08:51.601513 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:51.601458    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:51.601574 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:51.601552    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:52.762360 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:52.762322    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:52.762816 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:52.762440    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:52.991981 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:52.991949    2551 generic.go:356] "Generic (PLEG): container finished" podID="36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29" containerID="cbe9e073b72c9008034ce3aa40daafcb36dad83bd5030532dcdf770d6bbb03ee" exitCode=0
Sep 15 21:08:52.992158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:52.992035    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerDied","Data":"cbe9e073b72c9008034ce3aa40daafcb36dad83bd5030532dcdf770d6bbb03ee"}
Sep 15 21:08:53.762258 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:53.762223    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:53.762258 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:53.762255    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:53.762675 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:53.762349    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:53.762675 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:53.762461    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:54.762652 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:54.762616    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:54.762979 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:54.762720    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:55.000310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.000277    2551 generic.go:356] "Generic (PLEG): container finished" podID="36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29" containerID="d74a3b54516225db6cc32c7602fa77a7d008bc9961f774e3f50dad217799fb3b" exitCode=0
Sep 15 21:08:55.000469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.000331    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerDied","Data":"d74a3b54516225db6cc32c7602fa77a7d008bc9961f774e3f50dad217799fb3b"}
Sep 15 21:08:55.369669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.369639    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.369690    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.369727    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369793    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369826    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369841    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369857    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369862    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:09:27.369847206 +0000 UTC m=+66.171716130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369868    2551 projected.go:196] Error preparing data for projected volume kube-api-access-fq6wc for pod openshift-network-diagnostics/network-check-target-t89pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369886    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret podName:c1e92fdc-eaa5-48fa-9584-3eb7d009467c nodeName:}" failed. No retries permitted until 2026-09-15 21:09:27.369869889 +0000 UTC m=+66.171738813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret") pod "global-pull-secret-syncer-t22b7" (UID: "c1e92fdc-eaa5-48fa-9584-3eb7d009467c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:08:55.370006 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.369904    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc podName:f9249acd-262a-4669-8c40-cbae3ea027f1 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:27.369895148 +0000 UTC m=+66.171764072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq6wc" (UniqueName: "kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc") pod "network-check-target-t89pr" (UID: "f9249acd-262a-4669-8c40-cbae3ea027f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:08:55.763200 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.763165    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:55.763654 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.763313    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t89pr" podUID="f9249acd-262a-4669-8c40-cbae3ea027f1"
Sep 15 21:08:55.763654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:55.763377    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:55.763654 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:55.763503    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:08:56.762080 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:56.762044    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:56.762288 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:56.762157    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t22b7" podUID="c1e92fdc-eaa5-48fa-9584-3eb7d009467c"
Sep 15 21:08:57.446674 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.446476    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-188.ec2.internal" event="NodeReady"
Sep 15 21:08:57.447089 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.446789    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 21:08:57.510227 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.509344    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-42rkm"]
Sep 15 21:08:57.512895 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.512862    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c6bhh"]
Sep 15 21:08:57.513038 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.513002    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.516050 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.516027    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.520478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.520450    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p5jgl\""
Sep 15 21:08:57.522036 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.522014    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 21:08:57.522739 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.522571    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lthts\""
Sep 15 21:08:57.522842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.522795    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 21:08:57.523901 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.523748    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 21:08:57.524915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.524890    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 21:08:57.525822 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.525802    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 21:08:57.543225 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.543180    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6bhh"]
Sep 15 21:08:57.548318 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.547632    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-42rkm"]
Sep 15 21:08:57.689291 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.689248    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb7b\" (UniqueName: \"kubernetes.io/projected/7165655a-da5a-4d68-af9e-0ffbfa8589b3-kube-api-access-2sb7b\") pod \"ingress-canary-c6bhh\" (UID: \"7165655a-da5a-4d68-af9e-0ffbfa8589b3\") " pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.689483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.689356    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f0f048-6751-4841-abd2-323b67ae5e30-config-volume\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.689483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.689397    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7165655a-da5a-4d68-af9e-0ffbfa8589b3-cert\") pod \"ingress-canary-c6bhh\" (UID: \"7165655a-da5a-4d68-af9e-0ffbfa8589b3\") " pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.689483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.689422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drv8n\" (UniqueName: \"kubernetes.io/projected/e0f0f048-6751-4841-abd2-323b67ae5e30-kube-api-access-drv8n\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.689483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.689466    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.689692 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.689527    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f0f048-6751-4841-abd2-323b67ae5e30-tmp-dir\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.762585 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.762539    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:08:57.762783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.762758    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:08:57.765229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.765207    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:08:57.765641 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.765619    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lj4g\""
Sep 15 21:08:57.765821 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.765805    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:08:57.765992 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.765977    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bkm5f\""
Sep 15 21:08:57.766056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.765992    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:08:57.789833 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.789811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f0f048-6751-4841-abd2-323b67ae5e30-tmp-dir\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.789956 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.789843    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb7b\" (UniqueName: \"kubernetes.io/projected/7165655a-da5a-4d68-af9e-0ffbfa8589b3-kube-api-access-2sb7b\") pod \"ingress-canary-c6bhh\" (UID: \"7165655a-da5a-4d68-af9e-0ffbfa8589b3\") " pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.789956 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.789883    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f0f048-6751-4841-abd2-323b67ae5e30-config-volume\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.790058 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.790028    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7165655a-da5a-4d68-af9e-0ffbfa8589b3-cert\") pod \"ingress-canary-c6bhh\" (UID: \"7165655a-da5a-4d68-af9e-0ffbfa8589b3\") " pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.790109 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.790068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drv8n\" (UniqueName: \"kubernetes.io/projected/e0f0f048-6751-4841-abd2-323b67ae5e30-kube-api-access-drv8n\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.790109 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.790100    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.790234 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:57.790218    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:57.790234 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.790224    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f0f048-6751-4841-abd2-323b67ae5e30-tmp-dir\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.790347 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:57.790279    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:58.290260056 +0000 UTC m=+37.092129003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:57.790457 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.790438    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f0f048-6751-4841-abd2-323b67ae5e30-config-volume\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.794645 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.794624    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7165655a-da5a-4d68-af9e-0ffbfa8589b3-cert\") pod \"ingress-canary-c6bhh\" (UID: \"7165655a-da5a-4d68-af9e-0ffbfa8589b3\") " pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.801205 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.801164    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drv8n\" (UniqueName: \"kubernetes.io/projected/e0f0f048-6751-4841-abd2-323b67ae5e30-kube-api-access-drv8n\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:57.801366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.801349    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb7b\" (UniqueName: \"kubernetes.io/projected/7165655a-da5a-4d68-af9e-0ffbfa8589b3-kube-api-access-2sb7b\") pod \"ingress-canary-c6bhh\" (UID: \"7165655a-da5a-4d68-af9e-0ffbfa8589b3\") " pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:57.832355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:57.832325    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6bhh"
Sep 15 21:08:58.294934 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:58.294900    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:58.295135 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:58.295082    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:58.295221 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:58.295158    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:08:59.295140488 +0000 UTC m=+38.097009413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:08:58.763029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:58.762993    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:08:58.766079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:58.766057    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:08:59.286047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:59.286018    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6bhh"]
Sep 15 21:08:59.289341 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:08:59.289315    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7165655a_da5a_4d68_af9e_0ffbfa8589b3.slice/crio-2d6afb954c8643303a9459f5d84d6e48aec7470ebedd1c1848c556dfd73c557e WatchSource:0}: Error finding container 2d6afb954c8643303a9459f5d84d6e48aec7470ebedd1c1848c556dfd73c557e: Status 404 returned error can't find the container with id 2d6afb954c8643303a9459f5d84d6e48aec7470ebedd1c1848c556dfd73c557e
Sep 15 21:08:59.303913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:08:59.303888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:08:59.304072 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:59.304052    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:08:59.304149 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:08:59.304133    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:01.304108541 +0000 UTC m=+40.105977480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:00.017150 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:00.016555    2551 generic.go:356] "Generic (PLEG): container finished" podID="36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29" containerID="92dcba39822f177c2e314ab93e534eaf92a0908c78fdb1b9ef07981b8be55694" exitCode=0
Sep 15 21:09:00.017150 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:00.016642    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerDied","Data":"92dcba39822f177c2e314ab93e534eaf92a0908c78fdb1b9ef07981b8be55694"}
Sep 15 21:09:00.019763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:00.019737    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6bhh" event={"ID":"7165655a-da5a-4d68-af9e-0ffbfa8589b3","Type":"ContainerStarted","Data":"2d6afb954c8643303a9459f5d84d6e48aec7470ebedd1c1848c556dfd73c557e"}
Sep 15 21:09:01.027499 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:01.027464    2551 generic.go:356] "Generic (PLEG): container finished" podID="36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29" containerID="e68d27391c2627118acee9e6e666fb35c51f5f39742b0bf4a86bb413de8c756a" exitCode=0
Sep 15 21:09:01.027955 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:01.027545    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerDied","Data":"e68d27391c2627118acee9e6e666fb35c51f5f39742b0bf4a86bb413de8c756a"}
Sep 15 21:09:01.316827 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:01.316752    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:09:01.316962 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:01.316895    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:09:01.316962 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:01.316959    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:05.316942937 +0000 UTC m=+44.118811860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:02.033682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:02.033590    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" event={"ID":"36f5e3ee-6ca4-4799-81d8-bbd4e1cdfd29","Type":"ContainerStarted","Data":"54b31038e4e21cf390a56730e525c51de659ce0ed791b7cd27b731d695c6886f"}
Sep 15 21:09:02.035300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:02.035270    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6bhh" event={"ID":"7165655a-da5a-4d68-af9e-0ffbfa8589b3","Type":"ContainerStarted","Data":"51a9b7b660d218e036b5794b93de3eb4be797100541d5faf849604d4e4a995a5"}
Sep 15 21:09:02.057494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:02.057437    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r9f4b" podStartSLOduration=9.750441816 podStartE2EDuration="40.057420559s" totalImagesPullingTime="30.306978743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:02.056597914 +0000 UTC m=+40.858466877" watchObservedRunningTime="2026-09-15 21:09:02.057420559 +0000 UTC m=+40.859289502"
Sep 15 21:09:02.073813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:02.073760    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c6bhh" podStartSLOduration=2.491446652 podStartE2EDuration="5.073744807s" totalImagesPullingTime="2.582298155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:02.072756063 +0000 UTC m=+40.874625030" watchObservedRunningTime="2026-09-15 21:09:02.073744807 +0000 UTC m=+40.875613750"
Sep 15 21:09:05.347018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:05.346972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:09:05.347494 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:05.347126    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:09:05.347494 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:05.347212    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:13.347174015 +0000 UTC m=+52.149042958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:12.346698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.346665    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"]
Sep 15 21:09:12.365095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.365069    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"]
Sep 15 21:09:12.365286 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.365178    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.367553 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.367532    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 21:09:12.368621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.368599    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 21:09:12.368744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.368671    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 21:09:12.368744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.368685    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 21:09:12.393874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.393851    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"]
Sep 15 21:09:12.410744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.410718    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.410834 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.410783    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"]
Sep 15 21:09:12.415008 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.414818    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 21:09:12.415008 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.414846    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 21:09:12.415515 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.415496    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 21:09:12.415793 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.415772    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 21:09:12.494698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.494663    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e451f132-cf36-450a-a3b1-d815c65aaf25-tmp\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.494698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.494702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e451f132-cf36-450a-a3b1-d815c65aaf25-klusterlet-config\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.494919 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.494742    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzvs\" (UniqueName: \"kubernetes.io/projected/e451f132-cf36-450a-a3b1-d815c65aaf25-kube-api-access-pdzvs\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.595950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.595914    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.596132 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.595964    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-hub\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.596132 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.595988    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6nc\" (UniqueName: \"kubernetes.io/projected/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-kube-api-access-zd6nc\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.596132 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596058    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzvs\" (UniqueName: \"kubernetes.io/projected/e451f132-cf36-450a-a3b1-d815c65aaf25-kube-api-access-pdzvs\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.596303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596163    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e451f132-cf36-450a-a3b1-d815c65aaf25-tmp\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.596303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596215    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-ca\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.596303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596242    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e451f132-cf36-450a-a3b1-d815c65aaf25-klusterlet-config\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.596303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596257    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.596303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.596600 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.596580    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e451f132-cf36-450a-a3b1-d815c65aaf25-tmp\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.599582 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.599541    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e451f132-cf36-450a-a3b1-d815c65aaf25-klusterlet-config\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.605445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.605425    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzvs\" (UniqueName: \"kubernetes.io/projected/e451f132-cf36-450a-a3b1-d815c65aaf25-kube-api-access-pdzvs\") pod \"klusterlet-addon-workmgr-85d56d5b94-2qpzx\" (UID: \"e451f132-cf36-450a-a3b1-d815c65aaf25\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.674460 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.674425    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:12.697107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.697085    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-ca\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.697223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.697127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.697223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.697154    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.697223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.697181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.697382 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.697248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-hub\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.697382 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.697272    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6nc\" (UniqueName: \"kubernetes.io/projected/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-kube-api-access-zd6nc\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.698211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.698169    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.699850 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.699820    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.699942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.699854    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.699942 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.699862    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-ca\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.700052 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.700031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-hub\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.708109 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.708086    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6nc\" (UniqueName: \"kubernetes.io/projected/e23db1a8-c84f-45c8-97a6-7e5aad33e8f0-kube-api-access-zd6nc\") pod \"cluster-proxy-proxy-agent-748984d747-bnhvv\" (UID: \"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.732042 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.732009    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:09:12.967517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:12.967484    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"]
Sep 15 21:09:12.970777 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:09:12.970746    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode451f132_cf36_450a_a3b1_d815c65aaf25.slice/crio-b0d1da5af4927d65fbba76adf04a535e3af568991b1dea1f1fdbc0eea51cfcc7 WatchSource:0}: Error finding container b0d1da5af4927d65fbba76adf04a535e3af568991b1dea1f1fdbc0eea51cfcc7: Status 404 returned error can't find the container with id b0d1da5af4927d65fbba76adf04a535e3af568991b1dea1f1fdbc0eea51cfcc7
Sep 15 21:09:13.043419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:13.043391    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"]
Sep 15 21:09:13.047732 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:09:13.047707    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode23db1a8_c84f_45c8_97a6_7e5aad33e8f0.slice/crio-2ac4368e41e0eac9e2623d65f02aa044414e9ae4a9a65176a63728bf9900706e WatchSource:0}: Error finding container 2ac4368e41e0eac9e2623d65f02aa044414e9ae4a9a65176a63728bf9900706e: Status 404 returned error can't find the container with id 2ac4368e41e0eac9e2623d65f02aa044414e9ae4a9a65176a63728bf9900706e
Sep 15 21:09:13.062964 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:13.062930    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" event={"ID":"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0","Type":"ContainerStarted","Data":"2ac4368e41e0eac9e2623d65f02aa044414e9ae4a9a65176a63728bf9900706e"}
Sep 15 21:09:13.064110 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:13.064083    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx" event={"ID":"e451f132-cf36-450a-a3b1-d815c65aaf25","Type":"ContainerStarted","Data":"b0d1da5af4927d65fbba76adf04a535e3af568991b1dea1f1fdbc0eea51cfcc7"}
Sep 15 21:09:13.401681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:13.401588    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:09:13.402135 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:13.401735    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:09:13.402135 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:13.401805    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:09:29.401785591 +0000 UTC m=+68.203654512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:19.081924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.081888    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" event={"ID":"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0","Type":"ContainerStarted","Data":"531e27fbed8e7938f67bf8acb446a426e89960a58606ded9ad5a45d600e639f9"}
Sep 15 21:09:19.082444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.081936    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" event={"ID":"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0","Type":"ContainerStarted","Data":"3c751f811c42eb5b4e3fc60dcaf0637130779ac86251e16b7765c05d5b92bae3"}
Sep 15 21:09:19.082444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.081955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" event={"ID":"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0","Type":"ContainerStarted","Data":"1d44de6bc75774d6e1c74ad0e96570c2aadbca5d6692d20d8769ffdb217e07ba"}
Sep 15 21:09:19.083342 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.083315    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx" event={"ID":"e451f132-cf36-450a-a3b1-d815c65aaf25","Type":"ContainerStarted","Data":"bf8e5758f6cd51b11bf41b579fb3b2f65a73ff0cc7a81a9eb70397e06d6a63ae"}
Sep 15 21:09:19.083552 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.083536    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:19.085159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.085140    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:09:19.100910 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.100868    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" podStartSLOduration=1.574562211 podStartE2EDuration="7.100857251s" totalImagesPullingTime="5.52629504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:19.099944313 +0000 UTC m=+57.901813257" watchObservedRunningTime="2026-09-15 21:09:19.100857251 +0000 UTC m=+57.902726190"
Sep 15 21:09:19.114978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:19.114942    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx" podStartSLOduration=1.513996819 podStartE2EDuration="7.114928841s" totalImagesPullingTime="5.600932022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:19.114225176 +0000 UTC m=+57.916094133" watchObservedRunningTime="2026-09-15 21:09:19.114928841 +0000 UTC m=+57.916797784"
Sep 15 21:09:23.003217 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:23.003177    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qnx2"
Sep 15 21:09:27.407939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.407899    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:09:27.408467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.407949    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:09:27.408467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.408002    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:09:27.410598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.410562    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:09:27.410755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.410631    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:09:27.410755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.410638    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:09:27.418145 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:27.418123    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:09:27.418319 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:27.418233    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:10:31.418212542 +0000 UTC m=+130.220081476 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : secret "metrics-daemon-secret" not found
Sep 15 21:09:27.420254 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.420238    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:09:27.420688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.420671    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1e92fdc-eaa5-48fa-9584-3eb7d009467c-original-pull-secret\") pod \"global-pull-secret-syncer-t22b7\" (UID: \"c1e92fdc-eaa5-48fa-9584-3eb7d009467c\") " pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:09:27.431155 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.431133    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6wc\" (UniqueName: \"kubernetes.io/projected/f9249acd-262a-4669-8c40-cbae3ea027f1-kube-api-access-fq6wc\") pod \"network-check-target-t89pr\" (UID: \"f9249acd-262a-4669-8c40-cbae3ea027f1\") " pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:09:27.484468 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.484432    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bkm5f\""
Sep 15 21:09:27.492351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.492327    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:09:27.573722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.573694    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t22b7"
Sep 15 21:09:27.761802 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.761719    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t89pr"]
Sep 15 21:09:27.765165 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:09:27.765140    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9249acd_262a_4669_8c40_cbae3ea027f1.slice/crio-19f1d023bcf6f10f44f721e7d8e2c1ac1a659def499f6d69cc2c105fa422c653 WatchSource:0}: Error finding container 19f1d023bcf6f10f44f721e7d8e2c1ac1a659def499f6d69cc2c105fa422c653: Status 404 returned error can't find the container with id 19f1d023bcf6f10f44f721e7d8e2c1ac1a659def499f6d69cc2c105fa422c653
Sep 15 21:09:27.841571 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:27.841541    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t22b7"]
Sep 15 21:09:27.844813 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:09:27.844788    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e92fdc_eaa5_48fa_9584_3eb7d009467c.slice/crio-d83c4323a72b62a7f8f3c57cb89f1584ff01c908fae4dad8304f033bc1d08fd4 WatchSource:0}: Error finding container d83c4323a72b62a7f8f3c57cb89f1584ff01c908fae4dad8304f033bc1d08fd4: Status 404 returned error can't find the container with id d83c4323a72b62a7f8f3c57cb89f1584ff01c908fae4dad8304f033bc1d08fd4
Sep 15 21:09:28.106868 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:28.106773    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t22b7" event={"ID":"c1e92fdc-eaa5-48fa-9584-3eb7d009467c","Type":"ContainerStarted","Data":"d83c4323a72b62a7f8f3c57cb89f1584ff01c908fae4dad8304f033bc1d08fd4"}
Sep 15 21:09:28.107935 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:28.107910    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t89pr" event={"ID":"f9249acd-262a-4669-8c40-cbae3ea027f1","Type":"ContainerStarted","Data":"19f1d023bcf6f10f44f721e7d8e2c1ac1a659def499f6d69cc2c105fa422c653"}
Sep 15 21:09:29.423355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:29.423311    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:09:29.423767 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:29.423488    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:09:29.423767 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:09:29.423571    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:10:01.423555106 +0000 UTC m=+100.225424026 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:09:33.126358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:33.126310    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t89pr" event={"ID":"f9249acd-262a-4669-8c40-cbae3ea027f1","Type":"ContainerStarted","Data":"033674e8be0da82e95b070dd8947e6b78e71ecf189f82f7a580f79715097a20f"}
Sep 15 21:09:33.126812 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:33.126479    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:09:33.127844 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:33.127819    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t22b7" event={"ID":"c1e92fdc-eaa5-48fa-9584-3eb7d009467c","Type":"ContainerStarted","Data":"91f83c68a705cc9cbcf243f1bafad990d59283adf0f49757f16600b917d471a5"}
Sep 15 21:09:33.147726 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:33.147681    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t89pr" podStartSLOduration=66.666945686 podStartE2EDuration="1m11.147663917s" totalImagesPullingTime="4.480718231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:33.147082005 +0000 UTC m=+71.948950966" watchObservedRunningTime="2026-09-15 21:09:33.147663917 +0000 UTC m=+71.949532859"
Sep 15 21:09:33.162926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:09:33.162886    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t22b7" podStartSLOduration=66.758000361 podStartE2EDuration="1m11.162873187s" totalImagesPullingTime="4.404872826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:33.162437929 +0000 UTC m=+71.964306872" watchObservedRunningTime="2026-09-15 21:09:33.162873187 +0000 UTC m=+71.964742128"
Sep 15 21:10:01.443754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:10:01.443722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:10:01.444211 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:10:01.443862    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:10:01.444211 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:10:01.443926    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:11:05.443908941 +0000 UTC m=+164.245777866 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:10:04.133708 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:10:04.133673    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t89pr"
Sep 15 21:10:31.468038 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:10:31.467996    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:10:31.468503 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:10:31.468143    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:10:31.468503 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:10:31.468233    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:12:33.468213246 +0000 UTC m=+252.270082188 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : secret "metrics-daemon-secret" not found
Sep 15 21:11:00.527324 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:11:00.527273    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-42rkm" podUID="e0f0f048-6751-4841-abd2-323b67ae5e30"
Sep 15 21:11:00.775290 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:11:00.775252    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:11:01.381139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:01.381109    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42rkm"
Sep 15 21:11:05.508328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:05.508279    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:11:05.508772 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:11:05.508469    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 21:11:05.508772 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:11:05.508541    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls podName:e0f0f048-6751-4841-abd2-323b67ae5e30 nodeName:}" failed. No retries permitted until 2026-09-15 21:13:07.508525942 +0000 UTC m=+286.310394863 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls") pod "dns-default-42rkm" (UID: "e0f0f048-6751-4841-abd2-323b67ae5e30") : secret "dns-default-metrics-tls" not found
Sep 15 21:11:11.764688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:11.764650    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:11:19.083895 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:19.083795    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx" podUID="e451f132-cf36-450a-a3b1-d815c65aaf25" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Sep 15 21:11:19.436041 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:19.436009    2551 generic.go:356] "Generic (PLEG): container finished" podID="e451f132-cf36-450a-a3b1-d815c65aaf25" containerID="bf8e5758f6cd51b11bf41b579fb3b2f65a73ff0cc7a81a9eb70397e06d6a63ae" exitCode=1
Sep 15 21:11:19.436261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:19.436056    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx" event={"ID":"e451f132-cf36-450a-a3b1-d815c65aaf25","Type":"ContainerDied","Data":"bf8e5758f6cd51b11bf41b579fb3b2f65a73ff0cc7a81a9eb70397e06d6a63ae"}
Sep 15 21:11:19.436463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:19.436447    2551 scope.go:117] "RemoveContainer" containerID="bf8e5758f6cd51b11bf41b579fb3b2f65a73ff0cc7a81a9eb70397e06d6a63ae"
Sep 15 21:11:20.444222 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:20.441241    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx" event={"ID":"e451f132-cf36-450a-a3b1-d815c65aaf25","Type":"ContainerStarted","Data":"8290e1883bda63c3919fcccecd80b4ac20e319bb3dcbe8c7f58e13509778393c"}
Sep 15 21:11:20.444657 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:20.444363    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:11:21.443924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:21.443892    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:11:21.444512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:11:21.444491    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d56d5b94-2qpzx"
Sep 15 21:12:23.539762 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.539731    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q4zcm"]
Sep 15 21:12:23.541954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.541934    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.544010 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.543981    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 21:12:23.544122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.543991    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8r4hc\""
Sep 15 21:12:23.544767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.544747    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 21:12:23.544846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.544751    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 21:12:23.544846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.544780    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 21:12:23.555227 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.555184    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4zcm"]
Sep 15 21:12:23.652757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.652730    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.652915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.652762    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.652915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.652785    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfmg\" (UniqueName: \"kubernetes.io/projected/5ffde2dd-c294-46ad-940f-9815514bab78-kube-api-access-pqfmg\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.652915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.652881    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ffde2dd-c294-46ad-940f-9815514bab78-crio-socket\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.652915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.652909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffde2dd-c294-46ad-940f-9815514bab78-data-volume\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754046    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ffde2dd-c294-46ad-940f-9815514bab78-crio-socket\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754087    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffde2dd-c294-46ad-940f-9815514bab78-data-volume\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754132    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754165    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754173    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ffde2dd-c294-46ad-940f-9815514bab78-crio-socket\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754209    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfmg\" (UniqueName: \"kubernetes.io/projected/5ffde2dd-c294-46ad-940f-9815514bab78-kube-api-access-pqfmg\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.754454 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:23.754276    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:23.754454 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:23.754300    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:23.754454 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:23.754375    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:24.254353926 +0000 UTC m=+243.056222848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:23.754454 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:23.754401    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:24.254392992 +0000 UTC m=+243.056261919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:23.754597 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.754505    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffde2dd-c294-46ad-940f-9815514bab78-data-volume\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:23.762383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:23.762361    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfmg\" (UniqueName: \"kubernetes.io/projected/5ffde2dd-c294-46ad-940f-9815514bab78-kube-api-access-pqfmg\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:24.258083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:24.258049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:24.258264 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:24.258090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:24.258264 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:24.258170    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:24.258264 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:24.258216    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:24.258362 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:24.258272    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:25.258255113 +0000 UTC m=+244.060124033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:24.258362 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:24.258288    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:25.258280543 +0000 UTC m=+244.060149463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:25.265647 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:25.265612    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:25.266056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:25.265691    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:25.266056 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:25.265753    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:25.266056 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:25.265756    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:25.266056 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:25.265813    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:27.26579797 +0000 UTC m=+246.067666890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:25.266056 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:25.265824    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:27.265818719 +0000 UTC m=+246.067687640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:26.395095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:26.395068    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjwpf_05ad11b2-75b4-4a23-91cd-33be656bd8f6/dns-node-resolver/0.log"
Sep 15 21:12:27.277296 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:27.277256    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:27.277296 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:27.277298    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:27.277555 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:27.277402    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:27.277555 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:27.277412    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:27.277555 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:27.277466    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:31.277452458 +0000 UTC m=+250.079321392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:27.277555 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:27.277478    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:31.277472194 +0000 UTC m=+250.079341115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:27.595012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:27.594936    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p2hgv_7b9309ce-ed18-4f88-8b8e-28074c47e3f2/node-ca/0.log"
Sep 15 21:12:27.998202 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:27.998163    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6bhh_7165655a-da5a-4d68-af9e-0ffbfa8589b3/serve-healthcheck-canary/0.log"
Sep 15 21:12:31.308699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:31.308657    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:31.308699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:31.308704    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:31.309215 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:31.308787    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 21:12:31.309215 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:31.308815    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:31.309215 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:31.308862    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:39.308841904 +0000 UTC m=+258.110710891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : configmap "kube-rbac-proxy" not found
Sep 15 21:12:31.309215 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:31.308878    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls podName:5ffde2dd-c294-46ad-940f-9815514bab78 nodeName:}" failed. No retries permitted until 2026-09-15 21:12:39.308871711 +0000 UTC m=+258.110740631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q4zcm" (UID: "5ffde2dd-c294-46ad-940f-9815514bab78") : secret "insights-runtime-extractor-tls" not found
Sep 15 21:12:33.524180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:33.524141    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:12:33.524590 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:33.524268    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 21:12:33.524590 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:12:33.524324    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs podName:4f11c43a-261c-47bf-9fb4-842609d11c3e nodeName:}" failed. No retries permitted until 2026-09-15 21:14:35.524307511 +0000 UTC m=+374.326176436 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs") pod "network-metrics-daemon-kwhrg" (UID: "4f11c43a-261c-47bf-9fb4-842609d11c3e") : secret "metrics-daemon-secret" not found
Sep 15 21:12:39.371552 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:39.371512    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:39.371552 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:39.371556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:39.372161 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:39.372127    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ffde2dd-c294-46ad-940f-9815514bab78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:39.373837 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:39.373819    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ffde2dd-c294-46ad-940f-9815514bab78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q4zcm\" (UID: \"5ffde2dd-c294-46ad-940f-9815514bab78\") " pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:39.450630 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:39.450594    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q4zcm"
Sep 15 21:12:39.723128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:39.723088    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q4zcm"]
Sep 15 21:12:39.727900 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:12:39.727861    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffde2dd_c294_46ad_940f_9815514bab78.slice/crio-60d11f077b26b3711b5a229534bd9e8eafd218be184e3792589bd23fe8059fbc WatchSource:0}: Error finding container 60d11f077b26b3711b5a229534bd9e8eafd218be184e3792589bd23fe8059fbc: Status 404 returned error can't find the container with id 60d11f077b26b3711b5a229534bd9e8eafd218be184e3792589bd23fe8059fbc
Sep 15 21:12:40.671669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:40.671635    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4zcm" event={"ID":"5ffde2dd-c294-46ad-940f-9815514bab78","Type":"ContainerStarted","Data":"677bc5cc8013524e092c152dde8cd8840fec083abc780d60e52b88a5c8c7a669"}
Sep 15 21:12:40.672025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:40.671689    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4zcm" event={"ID":"5ffde2dd-c294-46ad-940f-9815514bab78","Type":"ContainerStarted","Data":"89d79245ad056054191e810e1ee39d915491d798bf101ff6cd06a6934bb87cfe"}
Sep 15 21:12:40.672025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:40.671708    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4zcm" event={"ID":"5ffde2dd-c294-46ad-940f-9815514bab78","Type":"ContainerStarted","Data":"60d11f077b26b3711b5a229534bd9e8eafd218be184e3792589bd23fe8059fbc"}
Sep 15 21:12:42.679304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:42.679227    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q4zcm" event={"ID":"5ffde2dd-c294-46ad-940f-9815514bab78","Type":"ContainerStarted","Data":"5c5f6adbe413ae21c810208d91449fe7de9afe0efbec5c348bd80b1424a7e292"}
Sep 15 21:12:42.695610 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:42.695568    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q4zcm" podStartSLOduration=17.734875643 podStartE2EDuration="19.695554389s" totalImagesPullingTime="1.960678746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:12:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:42.694594273 +0000 UTC m=+261.496463234" watchObservedRunningTime="2026-09-15 21:12:42.695554389 +0000 UTC m=+261.497423332"
Sep 15 21:12:45.657313 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.657282    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f4c7f9967-x6zfj"]
Sep 15 21:12:45.660344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.660327    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.663599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.663569    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 21:12:45.663721 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.663613    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 21:12:45.664445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.664433    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 21:12:45.664506 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.664471    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4hkpc\""
Sep 15 21:12:45.673103 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.673085    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 21:12:45.686910 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.686884    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f4c7f9967-x6zfj"]
Sep 15 21:12:45.813558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813522    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-bound-sa-token\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813552    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pln7j\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-kube-api-access-pln7j\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813573    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8ec35e3-4232-4790-bd19-ab4801c3176a-image-registry-private-configuration\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813594    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec35e3-4232-4790-bd19-ab4801c3176a-trusted-ca\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813644    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-registry-tls\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813684    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8ec35e3-4232-4790-bd19-ab4801c3176a-registry-certificates\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813714    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8ec35e3-4232-4790-bd19-ab4801c3176a-installation-pull-secrets\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.813763 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.813748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8ec35e3-4232-4790-bd19-ab4801c3176a-ca-trust-extracted\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.914944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.914861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8ec35e3-4232-4790-bd19-ab4801c3176a-registry-certificates\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.914944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.914901    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8ec35e3-4232-4790-bd19-ab4801c3176a-installation-pull-secrets\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.914944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.914939    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8ec35e3-4232-4790-bd19-ab4801c3176a-ca-trust-extracted\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.914989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-bound-sa-token\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915009    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pln7j\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-kube-api-access-pln7j\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8ec35e3-4232-4790-bd19-ab4801c3176a-image-registry-private-configuration\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915058    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec35e3-4232-4790-bd19-ab4801c3176a-trusted-ca\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915084    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-registry-tls\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915708    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8ec35e3-4232-4790-bd19-ab4801c3176a-ca-trust-extracted\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915967 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915805    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8ec35e3-4232-4790-bd19-ab4801c3176a-registry-certificates\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.915967 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.915948    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8ec35e3-4232-4790-bd19-ab4801c3176a-trusted-ca\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.917561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.917538    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8ec35e3-4232-4790-bd19-ab4801c3176a-installation-pull-secrets\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.917668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.917614    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8ec35e3-4232-4790-bd19-ab4801c3176a-image-registry-private-configuration\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.917782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.917763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-registry-tls\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.922971 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.922947    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-bound-sa-token\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.923399 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.923381    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pln7j\" (UniqueName: \"kubernetes.io/projected/a8ec35e3-4232-4790-bd19-ab4801c3176a-kube-api-access-pln7j\") pod \"image-registry-6f4c7f9967-x6zfj\" (UID: \"a8ec35e3-4232-4790-bd19-ab4801c3176a\") " pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:45.969317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:45.969296    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:46.240957 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:46.240881    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f4c7f9967-x6zfj"]
Sep 15 21:12:46.243605 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:12:46.243580    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ec35e3_4232_4790_bd19_ab4801c3176a.slice/crio-fab9c73e37350d9fe3cb2eeef888127e1ec242a0ceb72a67778a8a6027cd9bfd WatchSource:0}: Error finding container fab9c73e37350d9fe3cb2eeef888127e1ec242a0ceb72a67778a8a6027cd9bfd: Status 404 returned error can't find the container with id fab9c73e37350d9fe3cb2eeef888127e1ec242a0ceb72a67778a8a6027cd9bfd
Sep 15 21:12:46.692658 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:46.692623    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj" event={"ID":"a8ec35e3-4232-4790-bd19-ab4801c3176a","Type":"ContainerStarted","Data":"3590453ad568719f95dc029b55e8062d277fa7432b039772cdd322d8cb8e14cd"}
Sep 15 21:12:46.693054 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:46.692670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj" event={"ID":"a8ec35e3-4232-4790-bd19-ab4801c3176a","Type":"ContainerStarted","Data":"fab9c73e37350d9fe3cb2eeef888127e1ec242a0ceb72a67778a8a6027cd9bfd"}
Sep 15 21:12:46.693054 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:46.692739    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:12:46.710480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:12:46.710415    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj" podStartSLOduration=1.710398027 podStartE2EDuration="1.710398027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:46.709222523 +0000 UTC m=+265.511091492" watchObservedRunningTime="2026-09-15 21:12:46.710398027 +0000 UTC m=+265.512266971"
Sep 15 21:13:02.030380 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.030345    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n6xhr"]
Sep 15 21:13:02.035320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.035289    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.039107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.039082    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 21:13:02.039107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.039102    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 21:13:02.039329 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.039086    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 21:13:02.040229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.040214    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lwmf4\""
Sep 15 21:13:02.040386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.040375    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 21:13:02.040826 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.040809    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 21:13:02.040923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.040874    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 21:13:02.127738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127703    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-textfile\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.127915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127758    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xrn\" (UniqueName: \"kubernetes.io/projected/556a89df-43a5-4f4c-95af-5dda70c10295-kube-api-access-k9xrn\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.127915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127784    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-root\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.127915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127858    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-tls\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.127915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127895    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-sys\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.128081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/556a89df-43a5-4f4c-95af-5dda70c10295-metrics-client-ca\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.128081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127968    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-wtmp\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.128081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.127990    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.128081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.128056    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229180    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-tls\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229237    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-sys\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/556a89df-43a5-4f4c-95af-5dda70c10295-metrics-client-ca\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229275    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-wtmp\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229289    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229314    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-textfile\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229416    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-sys\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229456    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xrn\" (UniqueName: \"kubernetes.io/projected/556a89df-43a5-4f4c-95af-5dda70c10295-kube-api-access-k9xrn\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229475    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-wtmp\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-root\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/556a89df-43a5-4f4c-95af-5dda70c10295-root\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229873 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229741    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-textfile\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.229970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.229951    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/556a89df-43a5-4f4c-95af-5dda70c10295-metrics-client-ca\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.230262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.230243    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.231758 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.231733    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-tls\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.231758 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.231746    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/556a89df-43a5-4f4c-95af-5dda70c10295-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.242068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.242045    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xrn\" (UniqueName: \"kubernetes.io/projected/556a89df-43a5-4f4c-95af-5dda70c10295-kube-api-access-k9xrn\") pod \"node-exporter-n6xhr\" (UID: \"556a89df-43a5-4f4c-95af-5dda70c10295\") " pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.344473 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.344398    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n6xhr"
Sep 15 21:13:02.355503 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:13:02.355479    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556a89df_43a5_4f4c_95af_5dda70c10295.slice/crio-8ff8f09a48689c693db76dd5fe4648a3816caf427fd5a932c7ddab0b282e8f65 WatchSource:0}: Error finding container 8ff8f09a48689c693db76dd5fe4648a3816caf427fd5a932c7ddab0b282e8f65: Status 404 returned error can't find the container with id 8ff8f09a48689c693db76dd5fe4648a3816caf427fd5a932c7ddab0b282e8f65
Sep 15 21:13:02.733712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.733675    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" podUID="e23db1a8-c84f-45c8-97a6-7e5aad33e8f0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 21:13:02.740411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:02.740378    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6xhr" event={"ID":"556a89df-43a5-4f4c-95af-5dda70c10295","Type":"ContainerStarted","Data":"8ff8f09a48689c693db76dd5fe4648a3816caf427fd5a932c7ddab0b282e8f65"}
Sep 15 21:13:03.744924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:03.744887    2551 generic.go:356] "Generic (PLEG): container finished" podID="556a89df-43a5-4f4c-95af-5dda70c10295" containerID="21937a11b992b97423eb686b495c38781f2db62a5ff491aba5eb7afa9078a825" exitCode=0
Sep 15 21:13:03.745335 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:03.744951    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6xhr" event={"ID":"556a89df-43a5-4f4c-95af-5dda70c10295","Type":"ContainerDied","Data":"21937a11b992b97423eb686b495c38781f2db62a5ff491aba5eb7afa9078a825"}
Sep 15 21:13:04.381963 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:13:04.381906    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-42rkm" podUID="e0f0f048-6751-4841-abd2-323b67ae5e30"
Sep 15 21:13:04.749577 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:04.749548    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6xhr" event={"ID":"556a89df-43a5-4f4c-95af-5dda70c10295","Type":"ContainerStarted","Data":"5f4c348e76a526b71aaeb7071d9af03228005acc7b14625d9e34910d3c7cfde6"}
Sep 15 21:13:04.749960 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:04.749596    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6xhr" event={"ID":"556a89df-43a5-4f4c-95af-5dda70c10295","Type":"ContainerStarted","Data":"e416359ef8d9b00bc0dcc534728c2e17acf50d12b7d3119510f9230901cc0c29"}
Sep 15 21:13:04.749960 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:04.749603    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42rkm"
Sep 15 21:13:04.772310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:04.772269    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n6xhr" podStartSLOduration=2.82253846 podStartE2EDuration="3.772255067s" totalImagesPullingTime="880.714574ms" totalInitContainerRuntime="69.002033ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:04.771420871 +0000 UTC m=+283.573289857" watchObservedRunningTime="2026-09-15 21:13:04.772255067 +0000 UTC m=+283.574124011"
Sep 15 21:13:05.973662 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:05.973625    2551 patch_prober.go:28] interesting pod/image-registry-6f4c7f9967-x6zfj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 21:13:05.974043 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:05.973683    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj" podUID="a8ec35e3-4232-4790-bd19-ab4801c3176a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:13:07.565986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:07.565946    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:13:07.568384 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:07.568354    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0f0f048-6751-4841-abd2-323b67ae5e30-metrics-tls\") pod \"dns-default-42rkm\" (UID: \"e0f0f048-6751-4841-abd2-323b67ae5e30\") " pod="openshift-dns/dns-default-42rkm"
Sep 15 21:13:07.701681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:07.701654    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f4c7f9967-x6zfj"
Sep 15 21:13:07.753105 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:07.753076    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p5jgl\""
Sep 15 21:13:07.761360 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:07.761333    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42rkm"
Sep 15 21:13:08.037564 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:08.037529    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-42rkm"]
Sep 15 21:13:08.040913 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:13:08.040872    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f0f048_6751_4841_abd2_323b67ae5e30.slice/crio-9b0b875847c180e3ea82629a05feaa8671b0d37e0ae583494fabfb1e639fa3ef WatchSource:0}: Error finding container 9b0b875847c180e3ea82629a05feaa8671b0d37e0ae583494fabfb1e639fa3ef: Status 404 returned error can't find the container with id 9b0b875847c180e3ea82629a05feaa8671b0d37e0ae583494fabfb1e639fa3ef
Sep 15 21:13:08.762579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:08.762539    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42rkm" event={"ID":"e0f0f048-6751-4841-abd2-323b67ae5e30","Type":"ContainerStarted","Data":"9b0b875847c180e3ea82629a05feaa8671b0d37e0ae583494fabfb1e639fa3ef"}
Sep 15 21:13:10.770388 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:10.770352    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42rkm" event={"ID":"e0f0f048-6751-4841-abd2-323b67ae5e30","Type":"ContainerStarted","Data":"58b22a0290a6e3198fed6bdffc04f27665ffaa006a886b20f5eb0c5b4428d0a9"}
Sep 15 21:13:10.770812 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:10.770412    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42rkm" event={"ID":"e0f0f048-6751-4841-abd2-323b67ae5e30","Type":"ContainerStarted","Data":"dde24b5ae301958d92c149fa7f199f3b9bffd267e506372075f127fedf173475"}
Sep 15 21:13:10.770812 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:10.770508    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-42rkm"
Sep 15 21:13:10.792295 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:10.792245    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-42rkm" podStartSLOduration=252.041802286 podStartE2EDuration="4m13.792229036s" totalImagesPullingTime="1.75042675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:10.791375603 +0000 UTC m=+289.593244547" watchObservedRunningTime="2026-09-15 21:13:10.792229036 +0000 UTC m=+289.594097978"
Sep 15 21:13:12.733890 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:12.733852    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" podUID="e23db1a8-c84f-45c8-97a6-7e5aad33e8f0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 21:13:14.766144 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:13:14.766083    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kwhrg" podUID="4f11c43a-261c-47bf-9fb4-842609d11c3e"
Sep 15 21:13:20.776907 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:20.776874    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-42rkm"
Sep 15 21:13:21.617105 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:21.617082    2551 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:13:22.733362 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:22.733329    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" podUID="e23db1a8-c84f-45c8-97a6-7e5aad33e8f0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 21:13:22.738699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:22.733402    2551 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv"
Sep 15 21:13:22.738699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:22.734412    2551 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"531e27fbed8e7938f67bf8acb446a426e89960a58606ded9ad5a45d600e639f9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 15 21:13:22.738699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:22.734495    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" podUID="e23db1a8-c84f-45c8-97a6-7e5aad33e8f0" containerName="service-proxy" containerID="cri-o://531e27fbed8e7938f67bf8acb446a426e89960a58606ded9ad5a45d600e639f9" gracePeriod=30
Sep 15 21:13:23.813645 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:23.813614    2551 generic.go:356] "Generic (PLEG): container finished" podID="e23db1a8-c84f-45c8-97a6-7e5aad33e8f0" containerID="531e27fbed8e7938f67bf8acb446a426e89960a58606ded9ad5a45d600e639f9" exitCode=2
Sep 15 21:13:23.814042 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:23.813680    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" event={"ID":"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0","Type":"ContainerDied","Data":"531e27fbed8e7938f67bf8acb446a426e89960a58606ded9ad5a45d600e639f9"}
Sep 15 21:13:23.814042 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:23.813723    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-748984d747-bnhvv" event={"ID":"e23db1a8-c84f-45c8-97a6-7e5aad33e8f0","Type":"ContainerStarted","Data":"ab263863ccef57db7767c79eb3b7f20b21a0359b06de5736ca8e7763deb4e35e"}
Sep 15 21:13:25.762328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:25.762285    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:13:36.230738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.230697    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74fc9cb776-9fvpl"]
Sep 15 21:13:36.233700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.233683    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.235899 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.235874    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 21:13:36.236026 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.235949    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 21:13:36.236026 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.235994    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 21:13:36.236684 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.236666    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 21:13:36.236777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.236755    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g2wzw\""
Sep 15 21:13:36.236777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.236763    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 21:13:36.236885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.236807    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 21:13:36.236885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.236823    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 21:13:36.240416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.240401    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 21:13:36.244388 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.244370    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fc9cb776-9fvpl"]
Sep 15 21:13:36.366425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366378    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-trusted-ca-bundle\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.366425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366426    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-service-ca\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.366678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mf2\" (UniqueName: \"kubernetes.io/projected/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-kube-api-access-t5mf2\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.366678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366552    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-serving-cert\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.366678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366581    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-config\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.366678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366607    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-oauth-config\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.366678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.366642    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-oauth-serving-cert\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.467838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.467800    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mf2\" (UniqueName: \"kubernetes.io/projected/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-kube-api-access-t5mf2\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.467838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.467843    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-serving-cert\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.467861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-config\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.467877    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-oauth-config\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.467904    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-oauth-serving-cert\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.467989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-trusted-ca-bundle\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.468023    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-service-ca\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468684 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.468662    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-config\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468837 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.468816    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-service-ca\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.468938 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.468924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-trusted-ca-bundle\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.469008 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.468987    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-oauth-serving-cert\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.470549 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.470524    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-oauth-config\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.470666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.470645    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-serving-cert\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.476241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.476222    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mf2\" (UniqueName: \"kubernetes.io/projected/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-kube-api-access-t5mf2\") pod \"console-74fc9cb776-9fvpl\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") " pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.543543 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.543459    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:36.817740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.817658    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fc9cb776-9fvpl"]
Sep 15 21:13:36.821376 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:13:36.821348    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3050cbc8_8791_481e_b3f8_fbb4de5ca16f.slice/crio-ac726b66c60b4cf7f6dca9feda5fed84a34649ba2d15c7a0a5dd61102ec4313c WatchSource:0}: Error finding container ac726b66c60b4cf7f6dca9feda5fed84a34649ba2d15c7a0a5dd61102ec4313c: Status 404 returned error can't find the container with id ac726b66c60b4cf7f6dca9feda5fed84a34649ba2d15c7a0a5dd61102ec4313c
Sep 15 21:13:36.823657 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.823636    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:13:36.854310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:36.854279    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc9cb776-9fvpl" event={"ID":"3050cbc8-8791-481e-b3f8-fbb4de5ca16f","Type":"ContainerStarted","Data":"ac726b66c60b4cf7f6dca9feda5fed84a34649ba2d15c7a0a5dd61102ec4313c"}
Sep 15 21:13:39.865123 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:39.865032    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc9cb776-9fvpl" event={"ID":"3050cbc8-8791-481e-b3f8-fbb4de5ca16f","Type":"ContainerStarted","Data":"2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24"}
Sep 15 21:13:39.882494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:39.882443    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74fc9cb776-9fvpl" podStartSLOduration=1.226090779 podStartE2EDuration="3.882428601s" totalImagesPullingTime="2.656337822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:39.881343038 +0000 UTC m=+318.683212001" watchObservedRunningTime="2026-09-15 21:13:39.882428601 +0000 UTC m=+318.684297543"
Sep 15 21:13:46.544453 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:46.544406    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:46.544932 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:46.544503    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:46.549495 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:46.549474    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:46.892556 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:46.892469    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:13:49.045622 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:13:49.045593    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6bhh_7165655a-da5a-4d68-af9e-0ffbfa8589b3/serve-healthcheck-canary/0.log"
Sep 15 21:14:35.617324 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:35.617285    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:14:35.619707 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:35.619670    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f11c43a-261c-47bf-9fb4-842609d11c3e-metrics-certs\") pod \"network-metrics-daemon-kwhrg\" (UID: \"4f11c43a-261c-47bf-9fb4-842609d11c3e\") " pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:14:35.665845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:35.665821    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lj4g\""
Sep 15 21:14:35.673920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:35.673901    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwhrg"
Sep 15 21:14:35.951153 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:35.947246    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kwhrg"]
Sep 15 21:14:35.952804 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:14:35.952775    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f11c43a_261c_47bf_9fb4_842609d11c3e.slice/crio-a9991ffdb410550903d5528fcf017ae7c475309f77c121685cb17cb93e6adcf1 WatchSource:0}: Error finding container a9991ffdb410550903d5528fcf017ae7c475309f77c121685cb17cb93e6adcf1: Status 404 returned error can't find the container with id a9991ffdb410550903d5528fcf017ae7c475309f77c121685cb17cb93e6adcf1
Sep 15 21:14:36.050602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:36.050566    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kwhrg" event={"ID":"4f11c43a-261c-47bf-9fb4-842609d11c3e","Type":"ContainerStarted","Data":"a9991ffdb410550903d5528fcf017ae7c475309f77c121685cb17cb93e6adcf1"}
Sep 15 21:14:37.055535 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:37.055497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kwhrg" event={"ID":"4f11c43a-261c-47bf-9fb4-842609d11c3e","Type":"ContainerStarted","Data":"bb2f7e90d68470e6f021e4cfb6a8a3cea8fb7d88febed95ed0ca56d428264d79"}
Sep 15 21:14:37.055890 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:37.055553    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kwhrg" event={"ID":"4f11c43a-261c-47bf-9fb4-842609d11c3e","Type":"ContainerStarted","Data":"5afcc118321bec9017af0d221b5b7b209e811d23271621e305f16511f96ba472"}
Sep 15 21:14:37.072730 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:37.072603    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kwhrg" podStartSLOduration=374.213944192 podStartE2EDuration="6m15.072582359s" totalImagesPullingTime="858.638167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:37.072574411 +0000 UTC m=+375.874443371" watchObservedRunningTime="2026-09-15 21:14:37.072582359 +0000 UTC m=+375.874451318"
Sep 15 21:14:39.563174 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.563141    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74b55bf787-jzlxh"]
Sep 15 21:14:39.566230 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.566212    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.578912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.578890    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b55bf787-jzlxh"]
Sep 15 21:14:39.647328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647294    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-config\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.647328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647326    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-service-ca\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.647512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647348    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-trusted-ca-bundle\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.647512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647391    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-oauth-serving-cert\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.647512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647432    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tndt\" (UniqueName: \"kubernetes.io/projected/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-kube-api-access-8tndt\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.647512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647477    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-oauth-config\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.647512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.647504    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-serving-cert\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748567    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-oauth-config\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748597    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-serving-cert\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748626    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-config\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748639    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-service-ca\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748662    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-trusted-ca-bundle\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748676    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-oauth-serving-cert\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.748818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.748695    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tndt\" (UniqueName: \"kubernetes.io/projected/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-kube-api-access-8tndt\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.749454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.749425    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-oauth-serving-cert\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.749568 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.749462    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-service-ca\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.749639 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.749616    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-config\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.750073 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.750049    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-trusted-ca-bundle\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.751180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.751158    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-oauth-config\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.751413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.751392    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-serving-cert\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.756758 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.756738    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tndt\" (UniqueName: \"kubernetes.io/projected/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-kube-api-access-8tndt\") pod \"console-74b55bf787-jzlxh\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") " pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:39.875242 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:39.875147    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:40.144922 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:40.144852    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b55bf787-jzlxh"]
Sep 15 21:14:40.148275 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:14:40.148244    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb26d28_7baf_4cb6_b7f5_49941e1024e1.slice/crio-2e42e8e026edf119a93bcc65ccf16b0c6595e1a29e7b417d0a499f2430920712 WatchSource:0}: Error finding container 2e42e8e026edf119a93bcc65ccf16b0c6595e1a29e7b417d0a499f2430920712: Status 404 returned error can't find the container with id 2e42e8e026edf119a93bcc65ccf16b0c6595e1a29e7b417d0a499f2430920712
Sep 15 21:14:41.074181 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:41.074143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b55bf787-jzlxh" event={"ID":"0eb26d28-7baf-4cb6-b7f5-49941e1024e1","Type":"ContainerStarted","Data":"5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8"}
Sep 15 21:14:41.074553 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:41.074248    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b55bf787-jzlxh" event={"ID":"0eb26d28-7baf-4cb6-b7f5-49941e1024e1","Type":"ContainerStarted","Data":"2e42e8e026edf119a93bcc65ccf16b0c6595e1a29e7b417d0a499f2430920712"}
Sep 15 21:14:41.090865 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:41.090820    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74b55bf787-jzlxh" podStartSLOduration=2.090807124 podStartE2EDuration="2.090807124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:14:41.089745366 +0000 UTC m=+379.891614307" watchObservedRunningTime="2026-09-15 21:14:41.090807124 +0000 UTC m=+379.892676065"
Sep 15 21:14:49.875682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:49.875641    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:49.876078 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:49.875786    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:49.880437 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:49.880417    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:50.108240 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:50.108207    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:14:50.154867 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:14:50.154786    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fc9cb776-9fvpl"]
Sep 15 21:15:15.177060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.177021    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74fc9cb776-9fvpl" podUID="3050cbc8-8791-481e-b3f8-fbb4de5ca16f" containerName="console" containerID="cri-o://2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24" gracePeriod=15
Sep 15 21:15:15.417720 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.417698    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:15:15.502555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502467    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5mf2\" (UniqueName: \"kubernetes.io/projected/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-kube-api-access-t5mf2\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.502555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502523    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-serving-cert\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.502555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502549    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-service-ca\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.502888 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502568    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-trusted-ca-bundle\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.502888 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502584    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-oauth-config\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.502888 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502608    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-oauth-serving-cert\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.502888 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502671    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-config\") pod \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\" (UID: \"3050cbc8-8791-481e-b3f8-fbb4de5ca16f\") "
Sep 15 21:15:15.503076 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.502993    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:15.503076 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.503013    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:15.503076 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.503048    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:15.503183 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.503093    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-config" (OuterVolumeSpecName: "console-config") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:15:15.504966 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.504941    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:15.505205 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.505169    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-kube-api-access-t5mf2" (OuterVolumeSpecName: "kube-api-access-t5mf2") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "kube-api-access-t5mf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:15.505299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.505185    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3050cbc8-8791-481e-b3f8-fbb4de5ca16f" (UID: "3050cbc8-8791-481e-b3f8-fbb4de5ca16f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:15.604069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604016    2551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-config\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:15.604069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604063    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mf2\" (UniqueName: \"kubernetes.io/projected/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-kube-api-access-t5mf2\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:15.604069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604073    2551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-serving-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:15.604069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604082    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:15.604373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604091    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-trusted-ca-bundle\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:15.604373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604098    2551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-console-oauth-config\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:15.604373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:15.604106    2551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3050cbc8-8791-481e-b3f8-fbb4de5ca16f-oauth-serving-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:15:16.193610 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.193576    2551 generic.go:356] "Generic (PLEG): container finished" podID="3050cbc8-8791-481e-b3f8-fbb4de5ca16f" containerID="2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24" exitCode=0
Sep 15 21:15:16.193970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.193615    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fc9cb776-9fvpl"
Sep 15 21:15:16.193970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.193636    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc9cb776-9fvpl" event={"ID":"3050cbc8-8791-481e-b3f8-fbb4de5ca16f","Type":"ContainerDied","Data":"2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24"}
Sep 15 21:15:16.193970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.193682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fc9cb776-9fvpl" event={"ID":"3050cbc8-8791-481e-b3f8-fbb4de5ca16f","Type":"ContainerDied","Data":"ac726b66c60b4cf7f6dca9feda5fed84a34649ba2d15c7a0a5dd61102ec4313c"}
Sep 15 21:15:16.194422 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.194404    2551 scope.go:117] "RemoveContainer" containerID="2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24"
Sep 15 21:15:16.202474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.202458    2551 scope.go:117] "RemoveContainer" containerID="2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24"
Sep 15 21:15:16.202795 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:15:16.202776    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24\": container with ID starting with 2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24 not found: ID does not exist" containerID="2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24"
Sep 15 21:15:16.202837 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.202804    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24"} err="failed to get container status \"2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24\": rpc error: code = NotFound desc = could not find container \"2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24\": container with ID starting with 2571b7df2312276829740fbba27e9e037cee4324a25dd342f6646c6611ecaf24 not found: ID does not exist"
Sep 15 21:15:16.209371 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.209348    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fc9cb776-9fvpl"]
Sep 15 21:15:16.213163 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:16.213134    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74fc9cb776-9fvpl"]
Sep 15 21:15:17.766557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:17.766526    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3050cbc8-8791-481e-b3f8-fbb4de5ca16f" path="/var/lib/kubelet/pods/3050cbc8-8791-481e-b3f8-fbb4de5ca16f/volumes"
Sep 15 21:15:42.651583 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.651508    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"]
Sep 15 21:15:42.651954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.651726    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3050cbc8-8791-481e-b3f8-fbb4de5ca16f" containerName="console"
Sep 15 21:15:42.651954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.651736    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3050cbc8-8791-481e-b3f8-fbb4de5ca16f" containerName="console"
Sep 15 21:15:42.651954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.651778    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3050cbc8-8791-481e-b3f8-fbb4de5ca16f" containerName="console"
Sep 15 21:15:42.655399 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.655384    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.657536 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.657514    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 21:15:42.657728 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.657711    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:15:42.657925 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.657910    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2fnkh\""
Sep 15 21:15:42.668546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.668519    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"]
Sep 15 21:15:42.801971 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.801940    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzr5\" (UniqueName: \"kubernetes.io/projected/c476d789-b671-4a14-99ae-2cb8c214860c-kube-api-access-mgzr5\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pn6j\" (UID: \"c476d789-b671-4a14-99ae-2cb8c214860c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.802145 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.802022    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c476d789-b671-4a14-99ae-2cb8c214860c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pn6j\" (UID: \"c476d789-b671-4a14-99ae-2cb8c214860c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.902370 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.902305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c476d789-b671-4a14-99ae-2cb8c214860c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pn6j\" (UID: \"c476d789-b671-4a14-99ae-2cb8c214860c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.902370 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.902346    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzr5\" (UniqueName: \"kubernetes.io/projected/c476d789-b671-4a14-99ae-2cb8c214860c-kube-api-access-mgzr5\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pn6j\" (UID: \"c476d789-b671-4a14-99ae-2cb8c214860c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.902693 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.902675    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c476d789-b671-4a14-99ae-2cb8c214860c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pn6j\" (UID: \"c476d789-b671-4a14-99ae-2cb8c214860c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.912553 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.912525    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzr5\" (UniqueName: \"kubernetes.io/projected/c476d789-b671-4a14-99ae-2cb8c214860c-kube-api-access-mgzr5\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pn6j\" (UID: \"c476d789-b671-4a14-99ae-2cb8c214860c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:42.963997 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:42.963947    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"
Sep 15 21:15:43.245334 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:43.245301    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j"]
Sep 15 21:15:43.248538 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:15:43.248514    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc476d789_b671_4a14_99ae_2cb8c214860c.slice/crio-1da2badf82a6948a129efb413125aceb473c2129306ae543af577502cac317e9 WatchSource:0}: Error finding container 1da2badf82a6948a129efb413125aceb473c2129306ae543af577502cac317e9: Status 404 returned error can't find the container with id 1da2badf82a6948a129efb413125aceb473c2129306ae543af577502cac317e9
Sep 15 21:15:43.284711 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:43.284674    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j" event={"ID":"c476d789-b671-4a14-99ae-2cb8c214860c","Type":"ContainerStarted","Data":"1da2badf82a6948a129efb413125aceb473c2129306ae543af577502cac317e9"}
Sep 15 21:15:46.298928 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:46.298892    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j" event={"ID":"c476d789-b671-4a14-99ae-2cb8c214860c","Type":"ContainerStarted","Data":"df1070e9f300f488a645e2fc74069d9b421f0c0bae2531566744ceaa32eae710"}
Sep 15 21:15:46.330480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:15:46.326179    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pn6j" podStartSLOduration=1.7125351709999999 podStartE2EDuration="4.32615916s" totalImagesPullingTime="2.613623989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:46.324771381 +0000 UTC m=+445.126640332" watchObservedRunningTime="2026-09-15 21:15:46.32615916 +0000 UTC m=+445.128028104"
Sep 15 21:16:25.251473 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.251438    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"]
Sep 15 21:16:25.257167 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.257140    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.262268 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.262240    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:16:25.262588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.262570    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d2xlj\""
Sep 15 21:16:25.262687 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.262643    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:16:25.262687 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.262659    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:16:25.262687 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.262650    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:16:25.273873 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.273845    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"]
Sep 15 21:16:25.310738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.310706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/888d57b2-9405-4496-9bc6-b5c3701aac96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.310738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.310743    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888d57b2-9405-4496-9bc6-b5c3701aac96-tmp\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.310738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.310776    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/888d57b2-9405-4496-9bc6-b5c3701aac96-manifests\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.311357 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.310855    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/888d57b2-9405-4496-9bc6-b5c3701aac96-webhook-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.311357 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.310899    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtx8\" (UniqueName: \"kubernetes.io/projected/888d57b2-9405-4496-9bc6-b5c3701aac96-kube-api-access-nbtx8\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.411480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.411436    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/888d57b2-9405-4496-9bc6-b5c3701aac96-webhook-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.411480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.411482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtx8\" (UniqueName: \"kubernetes.io/projected/888d57b2-9405-4496-9bc6-b5c3701aac96-kube-api-access-nbtx8\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.411740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.411526    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/888d57b2-9405-4496-9bc6-b5c3701aac96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.411740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.411544    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888d57b2-9405-4496-9bc6-b5c3701aac96-tmp\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.411740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.411570    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/888d57b2-9405-4496-9bc6-b5c3701aac96-manifests\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.411919 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.411900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/888d57b2-9405-4496-9bc6-b5c3701aac96-manifests\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.414035 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.414010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888d57b2-9405-4496-9bc6-b5c3701aac96-tmp\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.414149 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.414086    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/888d57b2-9405-4496-9bc6-b5c3701aac96-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.414149 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.414124    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/888d57b2-9405-4496-9bc6-b5c3701aac96-webhook-cert\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.423927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.423902    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtx8\" (UniqueName: \"kubernetes.io/projected/888d57b2-9405-4496-9bc6-b5c3701aac96-kube-api-access-nbtx8\") pod \"opendatahub-operator-controller-manager-546ffd48db-r56ql\" (UID: \"888d57b2-9405-4496-9bc6-b5c3701aac96\") " pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.566733 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.566625    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:25.881211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:25.881168    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"]
Sep 15 21:16:25.882955 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:16:25.882926    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888d57b2_9405_4496_9bc6_b5c3701aac96.slice/crio-e4562c2aa5202f19ff4c47106530d4a80530805dbd71a1eb8fc6a69e2e45ffa4 WatchSource:0}: Error finding container e4562c2aa5202f19ff4c47106530d4a80530805dbd71a1eb8fc6a69e2e45ffa4: Status 404 returned error can't find the container with id e4562c2aa5202f19ff4c47106530d4a80530805dbd71a1eb8fc6a69e2e45ffa4
Sep 15 21:16:26.446172 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:26.446100    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql" event={"ID":"888d57b2-9405-4496-9bc6-b5c3701aac96","Type":"ContainerStarted","Data":"e4562c2aa5202f19ff4c47106530d4a80530805dbd71a1eb8fc6a69e2e45ffa4"}
Sep 15 21:16:29.459894 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:29.459856    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql" event={"ID":"888d57b2-9405-4496-9bc6-b5c3701aac96","Type":"ContainerStarted","Data":"090f699a9d700283103c638e0ac6b915185684f2bd31faa683759ebcea21e13b"}
Sep 15 21:16:30.466016 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:30.465982    2551 generic.go:356] "Generic (PLEG): container finished" podID="888d57b2-9405-4496-9bc6-b5c3701aac96" containerID="090f699a9d700283103c638e0ac6b915185684f2bd31faa683759ebcea21e13b" exitCode=0
Sep 15 21:16:30.466558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:30.466039    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql" event={"ID":"888d57b2-9405-4496-9bc6-b5c3701aac96","Type":"ContainerDied","Data":"090f699a9d700283103c638e0ac6b915185684f2bd31faa683759ebcea21e13b"}
Sep 15 21:16:31.471808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:31.471772    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql" event={"ID":"888d57b2-9405-4496-9bc6-b5c3701aac96","Type":"ContainerStarted","Data":"62a57c68180bb51decbeef2c30dc65ae0d68120f8f8b9cf82920fa75c07a1d00"}
Sep 15 21:16:31.472299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:31.471923    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:31.495805 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:31.495745    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql" podStartSLOduration=2.227993962 podStartE2EDuration="6.495727422s" totalImagesPullingTime="3.234345218s" totalInitContainerRuntime="1.033388242s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:31.494558742 +0000 UTC m=+490.296427717" watchObservedRunningTime="2026-09-15 21:16:31.495727422 +0000 UTC m=+490.297596364"
Sep 15 21:16:42.479185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:42.479155    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-546ffd48db-r56ql"
Sep 15 21:16:45.133428 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.133389    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"]
Sep 15 21:16:45.136574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.136557    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.143478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.143452    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 21:16:45.143595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.143519    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kcm72\""
Sep 15 21:16:45.143595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.143530    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 21:16:45.143595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.143538    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 21:16:45.143595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.143457    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:16:45.143801 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.143528    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 21:16:45.160860 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.160829    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"]
Sep 15 21:16:45.263270 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.263237    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/895c60fb-96d3-4ce1-ac49-2a7b102999c1-manager-config\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.263432 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.263296    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48q6c\" (UniqueName: \"kubernetes.io/projected/895c60fb-96d3-4ce1-ac49-2a7b102999c1-kube-api-access-48q6c\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.263432 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.263348    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/895c60fb-96d3-4ce1-ac49-2a7b102999c1-cert\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.263432 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.263392    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/895c60fb-96d3-4ce1-ac49-2a7b102999c1-metrics-cert\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.364273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.364236    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48q6c\" (UniqueName: \"kubernetes.io/projected/895c60fb-96d3-4ce1-ac49-2a7b102999c1-kube-api-access-48q6c\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.364273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.364275    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/895c60fb-96d3-4ce1-ac49-2a7b102999c1-cert\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.364483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.364295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/895c60fb-96d3-4ce1-ac49-2a7b102999c1-metrics-cert\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.364483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.364325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/895c60fb-96d3-4ce1-ac49-2a7b102999c1-manager-config\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.364955 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.364925    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/895c60fb-96d3-4ce1-ac49-2a7b102999c1-manager-config\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.366913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.366887    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/895c60fb-96d3-4ce1-ac49-2a7b102999c1-cert\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.367012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.366915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/895c60fb-96d3-4ce1-ac49-2a7b102999c1-metrics-cert\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.393080 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.393013    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48q6c\" (UniqueName: \"kubernetes.io/projected/895c60fb-96d3-4ce1-ac49-2a7b102999c1-kube-api-access-48q6c\") pod \"lws-controller-manager-68888cf94f-bjxlt\" (UID: \"895c60fb-96d3-4ce1-ac49-2a7b102999c1\") " pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.445997 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.445969    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:45.743386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:45.743354    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"]
Sep 15 21:16:45.746559 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:16:45.746533    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895c60fb_96d3_4ce1_ac49_2a7b102999c1.slice/crio-28fefbad5ead4d7419bb732d9879fb7c2ce407782710b0cdfc298d06c0ea9c3c WatchSource:0}: Error finding container 28fefbad5ead4d7419bb732d9879fb7c2ce407782710b0cdfc298d06c0ea9c3c: Status 404 returned error can't find the container with id 28fefbad5ead4d7419bb732d9879fb7c2ce407782710b0cdfc298d06c0ea9c3c
Sep 15 21:16:46.531794 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:46.531754    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt" event={"ID":"895c60fb-96d3-4ce1-ac49-2a7b102999c1","Type":"ContainerStarted","Data":"28fefbad5ead4d7419bb732d9879fb7c2ce407782710b0cdfc298d06c0ea9c3c"}
Sep 15 21:16:48.541769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:48.541733    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt" event={"ID":"895c60fb-96d3-4ce1-ac49-2a7b102999c1","Type":"ContainerStarted","Data":"dddd25c6728aecc3dfedaf06bf36be9860aa636235d60c9fbf8c5e9562a48878"}
Sep 15 21:16:48.542163 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:48.541844    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:48.559926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:48.559879    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt" podStartSLOduration=1.234403472 podStartE2EDuration="3.559862466s" totalImagesPullingTime="2.325458994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:48.558890803 +0000 UTC m=+507.360759746" watchObservedRunningTime="2026-09-15 21:16:48.559862466 +0000 UTC m=+507.361731409"
Sep 15 21:16:52.455409 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.455370    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"]
Sep 15 21:16:52.461623 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.461597    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.464228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.464206    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:16:52.464377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.464229    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:16:52.464377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.464249    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:16:52.464377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.464297    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g8cc4\""
Sep 15 21:16:52.475024 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.475000    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"]
Sep 15 21:16:52.624700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.624700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624701    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.624912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624720    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059792c3-8fc3-45bd-9977-59c02ca5818e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.624912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624814    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.624912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624854    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bx7\" (UniqueName: \"kubernetes.io/projected/059792c3-8fc3-45bd-9977-59c02ca5818e-kube-api-access-h8bx7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.625011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624937    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.625011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624964    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.625011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.624986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.625099 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.625012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726257 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726257 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059792c3-8fc3-45bd-9977-59c02ca5818e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726279    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726314    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bx7\" (UniqueName: \"kubernetes.io/projected/059792c3-8fc3-45bd-9977-59c02ca5818e-kube-api-access-h8bx7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726348    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726372    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726393    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726728 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726832 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726858    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.726938 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.726912    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.727020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.727003    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.727060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.727031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059792c3-8fc3-45bd-9977-59c02ca5818e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.729292 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.729270    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.729477 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.729459    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.734960 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.734896    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059792c3-8fc3-45bd-9977-59c02ca5818e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.735273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.735248    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bx7\" (UniqueName: \"kubernetes.io/projected/059792c3-8fc3-45bd-9977-59c02ca5818e-kube-api-access-h8bx7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq\" (UID: \"059792c3-8fc3-45bd-9977-59c02ca5818e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:52.772178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:52.772139    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:53.047857 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:53.047742    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"]
Sep 15 21:16:53.050835 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:16:53.050788    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059792c3_8fc3_45bd_9977_59c02ca5818e.slice/crio-13fd07622aafd331ae6b4b09518a15dd3ac522912566b6f046c81a9a8588ed76 WatchSource:0}: Error finding container 13fd07622aafd331ae6b4b09518a15dd3ac522912566b6f046c81a9a8588ed76: Status 404 returned error can't find the container with id 13fd07622aafd331ae6b4b09518a15dd3ac522912566b6f046c81a9a8588ed76
Sep 15 21:16:53.564363 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:53.564322    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq" event={"ID":"059792c3-8fc3-45bd-9977-59c02ca5818e","Type":"ContainerStarted","Data":"13fd07622aafd331ae6b4b09518a15dd3ac522912566b6f046c81a9a8588ed76"}
Sep 15 21:16:55.967636 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.967597    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:16:55.967962 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.967672    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:16:55.967962 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.967698    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:16:55.972625 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.972597    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c89ng"]
Sep 15 21:16:55.975791 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.975765    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:55.978741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.978716    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 15 21:16:55.978849 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.978757    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jrlzh\""
Sep 15 21:16:55.985479 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:55.985454    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c89ng"]
Sep 15 21:16:56.158693 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.158658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77skv\" (UniqueName: \"kubernetes.io/projected/cde5ef23-1fef-4673-b95c-27ef0a27575f-kube-api-access-77skv\") pod \"odh-model-controller-7998c67dc4-c89ng\" (UID: \"cde5ef23-1fef-4673-b95c-27ef0a27575f\") " pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.158874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.158821    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cde5ef23-1fef-4673-b95c-27ef0a27575f-cert\") pod \"odh-model-controller-7998c67dc4-c89ng\" (UID: \"cde5ef23-1fef-4673-b95c-27ef0a27575f\") " pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.259746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.259661    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cde5ef23-1fef-4673-b95c-27ef0a27575f-cert\") pod \"odh-model-controller-7998c67dc4-c89ng\" (UID: \"cde5ef23-1fef-4673-b95c-27ef0a27575f\") " pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.259746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.259699    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77skv\" (UniqueName: \"kubernetes.io/projected/cde5ef23-1fef-4673-b95c-27ef0a27575f-kube-api-access-77skv\") pod \"odh-model-controller-7998c67dc4-c89ng\" (UID: \"cde5ef23-1fef-4673-b95c-27ef0a27575f\") " pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.262327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.262302    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cde5ef23-1fef-4673-b95c-27ef0a27575f-cert\") pod \"odh-model-controller-7998c67dc4-c89ng\" (UID: \"cde5ef23-1fef-4673-b95c-27ef0a27575f\") " pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.279100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.279073    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77skv\" (UniqueName: \"kubernetes.io/projected/cde5ef23-1fef-4673-b95c-27ef0a27575f-kube-api-access-77skv\") pod \"odh-model-controller-7998c67dc4-c89ng\" (UID: \"cde5ef23-1fef-4673-b95c-27ef0a27575f\") " pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.287927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.287905    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:16:56.558040 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.557961    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k7t5c"]
Sep 15 21:16:56.562328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.562304    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.567082 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.567061    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 15 21:16:56.578579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.578547    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq" event={"ID":"059792c3-8fc3-45bd-9977-59c02ca5818e","Type":"ContainerStarted","Data":"d08159c643db5f6b5622dd5823ef444c8337b2a1c112259bd527461f47457492"}
Sep 15 21:16:56.585525 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.585499    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c89ng"]
Sep 15 21:16:56.586898 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.586868    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k7t5c"]
Sep 15 21:16:56.590301 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:16:56.590271    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde5ef23_1fef_4673_b95c_27ef0a27575f.slice/crio-2a188fcc352e1664ae64fcba4d32ec94b0e12758b46425331730ab124c85c24e WatchSource:0}: Error finding container 2a188fcc352e1664ae64fcba4d32ec94b0e12758b46425331730ab124c85c24e: Status 404 returned error can't find the container with id 2a188fcc352e1664ae64fcba4d32ec94b0e12758b46425331730ab124c85c24e
Sep 15 21:16:56.652261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.652180    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq" podStartSLOduration=1.737523478 podStartE2EDuration="4.652149064s" totalImagesPullingTime="2.914625586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:56.651220031 +0000 UTC m=+515.453088991" watchObservedRunningTime="2026-09-15 21:16:56.652149064 +0000 UTC m=+515.454018008"
Sep 15 21:16:56.663169 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.663136    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01370159-7c02-4486-8ded-4133a68fd3ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-k7t5c\" (UID: \"01370159-7c02-4486-8ded-4133a68fd3ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.663169 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.663170    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdc56\" (UniqueName: \"kubernetes.io/projected/01370159-7c02-4486-8ded-4133a68fd3ea-kube-api-access-rdc56\") pod \"model-serving-api-54d89bd5fc-k7t5c\" (UID: \"01370159-7c02-4486-8ded-4133a68fd3ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.764027 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.763987    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01370159-7c02-4486-8ded-4133a68fd3ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-k7t5c\" (UID: \"01370159-7c02-4486-8ded-4133a68fd3ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.764027 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.764029    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdc56\" (UniqueName: \"kubernetes.io/projected/01370159-7c02-4486-8ded-4133a68fd3ea-kube-api-access-rdc56\") pod \"model-serving-api-54d89bd5fc-k7t5c\" (UID: \"01370159-7c02-4486-8ded-4133a68fd3ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.766561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.766537    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01370159-7c02-4486-8ded-4133a68fd3ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-k7t5c\" (UID: \"01370159-7c02-4486-8ded-4133a68fd3ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.772085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.772063    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdc56\" (UniqueName: \"kubernetes.io/projected/01370159-7c02-4486-8ded-4133a68fd3ea-kube-api-access-rdc56\") pod \"model-serving-api-54d89bd5fc-k7t5c\" (UID: \"01370159-7c02-4486-8ded-4133a68fd3ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:56.772827 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.772814    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:56.871989 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:56.871903    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:57.145161 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:57.145088    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k7t5c"]
Sep 15 21:16:57.148065 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:16:57.148035    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01370159_7c02_4486_8ded_4133a68fd3ea.slice/crio-838d9ca53b2a85e71e7c97503ce5c37259f2b7c1e4430271f14da555ee8db99f WatchSource:0}: Error finding container 838d9ca53b2a85e71e7c97503ce5c37259f2b7c1e4430271f14da555ee8db99f: Status 404 returned error can't find the container with id 838d9ca53b2a85e71e7c97503ce5c37259f2b7c1e4430271f14da555ee8db99f
Sep 15 21:16:57.583667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:57.583632    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c" event={"ID":"01370159-7c02-4486-8ded-4133a68fd3ea","Type":"ContainerStarted","Data":"838d9ca53b2a85e71e7c97503ce5c37259f2b7c1e4430271f14da555ee8db99f"}
Sep 15 21:16:57.584747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:57.584719    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng" event={"ID":"cde5ef23-1fef-4673-b95c-27ef0a27575f","Type":"ContainerStarted","Data":"2a188fcc352e1664ae64fcba4d32ec94b0e12758b46425331730ab124c85c24e"}
Sep 15 21:16:57.778249 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:57.778220    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:58.590083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:58.590051    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:58.591394 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:58.591360    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq"
Sep 15 21:16:59.550542 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:59.550495    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68888cf94f-bjxlt"
Sep 15 21:16:59.597525 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:59.597482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c" event={"ID":"01370159-7c02-4486-8ded-4133a68fd3ea","Type":"ContainerStarted","Data":"77265a50fac476846dac2cbd7d0d02dfbfc096e7b3759d4c381f51e525fa6f8d"}
Sep 15 21:16:59.597990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:59.597596    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:16:59.617331 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:16:59.617218    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c" podStartSLOduration=1.405313137 podStartE2EDuration="3.617169612s" totalImagesPullingTime="2.211856475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:59.615003335 +0000 UTC m=+518.416872299" watchObservedRunningTime="2026-09-15 21:16:59.617169612 +0000 UTC m=+518.419038557"
Sep 15 21:17:01.608177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:01.608081    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng" event={"ID":"cde5ef23-1fef-4673-b95c-27ef0a27575f","Type":"ContainerStarted","Data":"c0084ce0721f03ed677204840356c4dec161d511aff463d4b09e319cc970a2d2"}
Sep 15 21:17:01.608584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:01.608211    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:17:01.636748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:01.636693    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng" podStartSLOduration=1.8734354149999999 podStartE2EDuration="6.636674598s" totalImagesPullingTime="4.763239183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:01.634912891 +0000 UTC m=+520.436781834" watchObservedRunningTime="2026-09-15 21:17:01.636674598 +0000 UTC m=+520.438543553"
Sep 15 21:17:06.359727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.359685    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"]
Sep 15 21:17:06.363016 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.362988    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.366145 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.366120    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 21:17:06.366145 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.366134    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bq2x2\""
Sep 15 21:17:06.375746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.375718    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"]
Sep 15 21:17:06.379180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.379155    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"]
Sep 15 21:17:06.379332 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.379269    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:06.381669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.381643    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j66jt\""
Sep 15 21:17:06.382328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.382314    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 21:17:06.399129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.399090    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"]
Sep 15 21:17:06.443401 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.443369    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkf6k\" (UniqueName: \"kubernetes.io/projected/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-kube-api-access-lkf6k\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:06.443584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.443422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:06.443584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.443480    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ede3f22-faa4-43fa-b430-4c37a8c417eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-8pzvp\" (UID: \"5ede3f22-faa4-43fa-b430-4c37a8c417eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.443584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.443509    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7pz4\" (UniqueName: \"kubernetes.io/projected/5ede3f22-faa4-43fa-b430-4c37a8c417eb-kube-api-access-x7pz4\") pod \"kserve-controller-manager-d6f4b65b8-8pzvp\" (UID: \"5ede3f22-faa4-43fa-b430-4c37a8c417eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.544697 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.544660    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkf6k\" (UniqueName: \"kubernetes.io/projected/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-kube-api-access-lkf6k\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:06.544866 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.544714    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:06.544866 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.544737    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ede3f22-faa4-43fa-b430-4c37a8c417eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-8pzvp\" (UID: \"5ede3f22-faa4-43fa-b430-4c37a8c417eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.544866 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.544755    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7pz4\" (UniqueName: \"kubernetes.io/projected/5ede3f22-faa4-43fa-b430-4c37a8c417eb-kube-api-access-x7pz4\") pod \"kserve-controller-manager-d6f4b65b8-8pzvp\" (UID: \"5ede3f22-faa4-43fa-b430-4c37a8c417eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.544964 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:17:06.544869    2551 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 15 21:17:06.544964 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:17:06.544937    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-cert podName:695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:07.044917524 +0000 UTC m=+525.846786455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-cert") pod "llmisvc-controller-manager-777c6bd7d5-g2lrz" (UID: "695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0") : secret "llmisvc-webhook-server-cert" not found
Sep 15 21:17:06.547300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.547278    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ede3f22-faa4-43fa-b430-4c37a8c417eb-cert\") pod \"kserve-controller-manager-d6f4b65b8-8pzvp\" (UID: \"5ede3f22-faa4-43fa-b430-4c37a8c417eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.555543 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.555520    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7pz4\" (UniqueName: \"kubernetes.io/projected/5ede3f22-faa4-43fa-b430-4c37a8c417eb-kube-api-access-x7pz4\") pod \"kserve-controller-manager-d6f4b65b8-8pzvp\" (UID: \"5ede3f22-faa4-43fa-b430-4c37a8c417eb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.557938 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.557915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkf6k\" (UniqueName: \"kubernetes.io/projected/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-kube-api-access-lkf6k\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:06.673008 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.672976    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:06.945573 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:06.945545    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"]
Sep 15 21:17:06.948183 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:17:06.948148    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ede3f22_faa4_43fa_b430_4c37a8c417eb.slice/crio-388e728e2b65135f62dfe3bf3a589a267989fa058f6f62163bac6638b006fade WatchSource:0}: Error finding container 388e728e2b65135f62dfe3bf3a589a267989fa058f6f62163bac6638b006fade: Status 404 returned error can't find the container with id 388e728e2b65135f62dfe3bf3a589a267989fa058f6f62163bac6638b006fade
Sep 15 21:17:07.049275 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:07.049235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:07.051757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:07.051728    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-g2lrz\" (UID: \"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:07.289614 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:07.289522    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:07.592080 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:07.591998    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"]
Sep 15 21:17:07.594636 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:17:07.594608    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod695dfc64_b0f6_47ea_91e7_deb2f0f4f8c0.slice/crio-9b617cb78a4f2e9fe583bd2761f87afb7485db0436aea33c59361d3d8a0f67d0 WatchSource:0}: Error finding container 9b617cb78a4f2e9fe583bd2761f87afb7485db0436aea33c59361d3d8a0f67d0: Status 404 returned error can't find the container with id 9b617cb78a4f2e9fe583bd2761f87afb7485db0436aea33c59361d3d8a0f67d0
Sep 15 21:17:07.633397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:07.633363    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz" event={"ID":"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0","Type":"ContainerStarted","Data":"9b617cb78a4f2e9fe583bd2761f87afb7485db0436aea33c59361d3d8a0f67d0"}
Sep 15 21:17:07.634505 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:07.634483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp" event={"ID":"5ede3f22-faa4-43fa-b430-4c37a8c417eb","Type":"ContainerStarted","Data":"388e728e2b65135f62dfe3bf3a589a267989fa058f6f62163bac6638b006fade"}
Sep 15 21:17:10.606990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:10.606912    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-k7t5c"
Sep 15 21:17:11.656455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:11.656416    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz" event={"ID":"695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0","Type":"ContainerStarted","Data":"bf27dba250f6e913553424e60ea94561d20165f6732165dab669d3086ba315a4"}
Sep 15 21:17:11.656931 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:11.656531    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:11.658009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:11.657988    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp" event={"ID":"5ede3f22-faa4-43fa-b430-4c37a8c417eb","Type":"ContainerStarted","Data":"b97ed4916a267e248e466c90a55d9b4ac12b045fad191bd77d1558d90c4a2040"}
Sep 15 21:17:11.658124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:11.658112    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:17:11.673480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:11.673433    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz" podStartSLOduration=2.161507745 podStartE2EDuration="5.673416946s" totalImagesPullingTime="3.511909201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:11.671695858 +0000 UTC m=+530.473564824" watchObservedRunningTime="2026-09-15 21:17:11.673416946 +0000 UTC m=+530.475285890"
Sep 15 21:17:11.688078 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:11.688029    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp" podStartSLOduration=2.161496848 podStartE2EDuration="5.688015175s" totalImagesPullingTime="3.526518327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:11.687372942 +0000 UTC m=+530.489241907" watchObservedRunningTime="2026-09-15 21:17:11.688015175 +0000 UTC m=+530.489884157"
Sep 15 21:17:12.615117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:12.615031    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-c89ng"
Sep 15 21:17:42.665082 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:42.665049    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-g2lrz"
Sep 15 21:17:42.668177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:17:42.668148    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8pzvp"
Sep 15 21:18:17.198248 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.198204    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"]
Sep 15 21:18:17.201577 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.201555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:17.204134 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.204112    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 21:18:17.204289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.204112    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:18:17.204446 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.204430    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:18:17.204936 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.204916    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6hqkc\""
Sep 15 21:18:17.220571 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.220542    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"]
Sep 15 21:18:17.239371 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.239338    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqdm\" (UniqueName: \"kubernetes.io/projected/95627f6b-0ef0-4535-b895-492dbcc68f2f-kube-api-access-2rqdm\") pod \"dns-operator-controller-manager-7b84cd7c9c-bqxrq\" (UID: \"95627f6b-0ef0-4535-b895-492dbcc68f2f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:17.340347 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.340303    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqdm\" (UniqueName: \"kubernetes.io/projected/95627f6b-0ef0-4535-b895-492dbcc68f2f-kube-api-access-2rqdm\") pod \"dns-operator-controller-manager-7b84cd7c9c-bqxrq\" (UID: \"95627f6b-0ef0-4535-b895-492dbcc68f2f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:17.350419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.350390    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqdm\" (UniqueName: \"kubernetes.io/projected/95627f6b-0ef0-4535-b895-492dbcc68f2f-kube-api-access-2rqdm\") pod \"dns-operator-controller-manager-7b84cd7c9c-bqxrq\" (UID: \"95627f6b-0ef0-4535-b895-492dbcc68f2f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:17.511897 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.511802    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:17.785884 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.785858    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"]
Sep 15 21:18:17.788555 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:18:17.788526    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95627f6b_0ef0_4535_b895_492dbcc68f2f.slice/crio-85e7ac801a43abc349e7d52ac98479b0de6de65e8d355b5578f344f455c6fb9a WatchSource:0}: Error finding container 85e7ac801a43abc349e7d52ac98479b0de6de65e8d355b5578f344f455c6fb9a: Status 404 returned error can't find the container with id 85e7ac801a43abc349e7d52ac98479b0de6de65e8d355b5578f344f455c6fb9a
Sep 15 21:18:17.927255 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:17.927218    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq" event={"ID":"95627f6b-0ef0-4535-b895-492dbcc68f2f","Type":"ContainerStarted","Data":"85e7ac801a43abc349e7d52ac98479b0de6de65e8d355b5578f344f455c6fb9a"}
Sep 15 21:18:20.942040 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:20.942003    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq" event={"ID":"95627f6b-0ef0-4535-b895-492dbcc68f2f","Type":"ContainerStarted","Data":"c422255f801b8d8517d92a884b4e0c2f37801966c82e31fe12f0a397ed589864"}
Sep 15 21:18:20.942461 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:20.942095    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:20.959719 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:20.959665    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq" podStartSLOduration=1.013433738 podStartE2EDuration="3.959647751s" totalImagesPullingTime="2.946214013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:20.958408382 +0000 UTC m=+599.760277348" watchObservedRunningTime="2026-09-15 21:18:20.959647751 +0000 UTC m=+599.761516693"
Sep 15 21:18:22.296496 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.296462    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"]
Sep 15 21:18:22.299795 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.299780    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:22.302178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.302154    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jngc9\""
Sep 15 21:18:22.317328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.317301    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"]
Sep 15 21:18:22.386367 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.386328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwv9z\" (UniqueName: \"kubernetes.io/projected/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0-kube-api-access-pwv9z\") pod \"limitador-operator-controller-manager-74495c69f8-bkh8q\" (UID: \"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:22.486904 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.486872    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwv9z\" (UniqueName: \"kubernetes.io/projected/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0-kube-api-access-pwv9z\") pod \"limitador-operator-controller-manager-74495c69f8-bkh8q\" (UID: \"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:22.495138 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.495106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwv9z\" (UniqueName: \"kubernetes.io/projected/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0-kube-api-access-pwv9z\") pod \"limitador-operator-controller-manager-74495c69f8-bkh8q\" (UID: \"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:22.609322 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.609239    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:22.887438 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.887258    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"]
Sep 15 21:18:22.895338 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:18:22.892377    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e8968b_6eed_4f15_a3b4_6df21cbf19a0.slice/crio-145953f80479f093cf41fc1ba472c23b6a205d1cf8bd7a28c88d36c42ef1bf88 WatchSource:0}: Error finding container 145953f80479f093cf41fc1ba472c23b6a205d1cf8bd7a28c88d36c42ef1bf88: Status 404 returned error can't find the container with id 145953f80479f093cf41fc1ba472c23b6a205d1cf8bd7a28c88d36c42ef1bf88
Sep 15 21:18:22.951806 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:22.951771    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q" event={"ID":"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0","Type":"ContainerStarted","Data":"145953f80479f093cf41fc1ba472c23b6a205d1cf8bd7a28c88d36c42ef1bf88"}
Sep 15 21:18:24.962720 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:24.962682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q" event={"ID":"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0","Type":"ContainerStarted","Data":"f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c"}
Sep 15 21:18:24.963120 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:24.962754    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:24.978434 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:24.978385    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q" podStartSLOduration=1.746577316 podStartE2EDuration="2.978369926s" totalImagesPullingTime="1.23179261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:24.977095991 +0000 UTC m=+603.778964946" watchObservedRunningTime="2026-09-15 21:18:24.978369926 +0000 UTC m=+603.780238868"
Sep 15 21:18:31.948404 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:31.948323    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bqxrq"
Sep 15 21:18:33.594135 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.594095    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"]
Sep 15 21:18:33.594552 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.594381    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" containerName="manager" containerID="cri-o://f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c" gracePeriod=2
Sep 15 21:18:33.596164 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.596129    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:33.603467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.603431    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"]
Sep 15 21:18:33.607849 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:18:33.607807    2551 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-bkh8q\": pods \"limitador-operator-controller-manager-74495c69f8-bkh8q\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:33.609761 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:18:33.609725    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bkh8q\" is forbidden: User \"system:node:ip-10-0-141-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-188.ec2.internal' and this object" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:33.831982 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.831955    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:33.986339 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.986304    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwv9z\" (UniqueName: \"kubernetes.io/projected/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0-kube-api-access-pwv9z\") pod \"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0\" (UID: \"a7e8968b-6eed-4f15-a3b4-6df21cbf19a0\") "
Sep 15 21:18:33.988676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:33.988643    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0-kube-api-access-pwv9z" (OuterVolumeSpecName: "kube-api-access-pwv9z") pod "a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" (UID: "a7e8968b-6eed-4f15-a3b4-6df21cbf19a0"). InnerVolumeSpecName "kube-api-access-pwv9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:34.003248 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:34.003216    2551 generic.go:356] "Generic (PLEG): container finished" podID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" containerID="f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c" exitCode=0
Sep 15 21:18:34.003417 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:34.003258    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:34.003417 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:34.003304    2551 scope.go:117] "RemoveContainer" containerID="f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c"
Sep 15 21:18:34.011924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:34.011904    2551 scope.go:117] "RemoveContainer" containerID="f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c"
Sep 15 21:18:34.012252 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:18:34.012234    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c\": container with ID starting with f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c not found: ID does not exist" containerID="f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c"
Sep 15 21:18:34.012304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:34.012261    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c"} err="failed to get container status \"f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c\": rpc error: code = NotFound desc = could not find container \"f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c\": container with ID starting with f46e26df868152e7f049fc29cc2f184ef3f85432e9d3c0f3f9550aeb47349e6c not found: ID does not exist"
Sep 15 21:18:34.014071 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:18:34.014042    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bkh8q\" is forbidden: User \"system:node:ip-10-0-141-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-188.ec2.internal' and this object" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bkh8q"
Sep 15 21:18:34.087028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:34.086991    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwv9z\" (UniqueName: \"kubernetes.io/projected/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0-kube-api-access-pwv9z\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:18:35.767637 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:35.767598    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" path="/var/lib/kubelet/pods/a7e8968b-6eed-4f15-a3b4-6df21cbf19a0/volumes"
Sep 15 21:18:39.656644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:18:39.656604    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b55bf787-jzlxh"]
Sep 15 21:19:03.851996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.851917    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"]
Sep 15 21:19:03.852543 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.852292    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" containerName="manager"
Sep 15 21:19:03.852543 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.852309    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" containerName="manager"
Sep 15 21:19:03.852543 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.852383    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e8968b-6eed-4f15-a3b4-6df21cbf19a0" containerName="manager"
Sep 15 21:19:03.862269 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.862232    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.864864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.864844    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xc9fn\""
Sep 15 21:19:03.869627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.869604    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"]
Sep 15 21:19:03.937108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937074    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wtm\" (UniqueName: \"kubernetes.io/projected/7aee3b2d-7f62-48a5-89cf-e0d93904115f-kube-api-access-x5wtm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937126    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937162    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937213    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937297    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937330    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937358    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:03.937471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:03.937418    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038378    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038511    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038557    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038576    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038611    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038648    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038679    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wtm\" (UniqueName: \"kubernetes.io/projected/7aee3b2d-7f62-48a5-89cf-e0d93904115f-kube-api-access-x5wtm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038799    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.038926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.038825    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.039039 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.039022    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.039092 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.039056    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.039397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.039342    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.040933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.040905    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.041126 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.041107    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.047013 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.046972    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aee3b2d-7f62-48a5-89cf-e0d93904115f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.047159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.047140    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wtm\" (UniqueName: \"kubernetes.io/projected/7aee3b2d-7f62-48a5-89cf-e0d93904115f-kube-api-access-x5wtm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-8xm2l\" (UID: \"7aee3b2d-7f62-48a5-89cf-e0d93904115f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.173962 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.173930    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:04.448542 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.448473    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"]
Sep 15 21:19:04.450835 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:19:04.450804    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aee3b2d_7f62_48a5_89cf_e0d93904115f.slice/crio-1ab03382d3a120e69ce4aa240fb4b91c39ab5f6b9f2a8c5bca60b1bb449c1b51 WatchSource:0}: Error finding container 1ab03382d3a120e69ce4aa240fb4b91c39ab5f6b9f2a8c5bca60b1bb449c1b51: Status 404 returned error can't find the container with id 1ab03382d3a120e69ce4aa240fb4b91c39ab5f6b9f2a8c5bca60b1bb449c1b51
Sep 15 21:19:04.453115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.453077    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:19:04.453180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.453170    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:19:04.453246 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.453221    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:19:04.676640 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.676583    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74b55bf787-jzlxh" podUID="0eb26d28-7baf-4cb6-b7f5-49941e1024e1" containerName="console" containerID="cri-o://5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8" gracePeriod=15
Sep 15 21:19:04.915114 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:04.915091    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:19:05.048967 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.048937    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-config\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.048986    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-trusted-ca-bundle\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049016    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-service-ca\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049035    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tndt\" (UniqueName: \"kubernetes.io/projected/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-kube-api-access-8tndt\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049084    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-serving-cert\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049113    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-oauth-config\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049135    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-oauth-serving-cert\") pod \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\" (UID: \"0eb26d28-7baf-4cb6-b7f5-49941e1024e1\") "
Sep 15 21:19:05.049517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049489    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-config" (OuterVolumeSpecName: "console-config") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:19:05.049573 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049536    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:19:05.049869 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049819    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:19:05.049949 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.049912    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:19:05.051669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.051646    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:19:05.052067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.052051    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-kube-api-access-8tndt" (OuterVolumeSpecName: "kube-api-access-8tndt") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "kube-api-access-8tndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:05.052067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.052052    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0eb26d28-7baf-4cb6-b7f5-49941e1024e1" (UID: "0eb26d28-7baf-4cb6-b7f5-49941e1024e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:19:05.135652 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.135611    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" event={"ID":"7aee3b2d-7f62-48a5-89cf-e0d93904115f","Type":"ContainerStarted","Data":"c187a4b1c5c15407b26bfd4877e3b6b28c8b6ce45c6800da17476237905ba03c"}
Sep 15 21:19:05.135846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.135684    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" event={"ID":"7aee3b2d-7f62-48a5-89cf-e0d93904115f","Type":"ContainerStarted","Data":"1ab03382d3a120e69ce4aa240fb4b91c39ab5f6b9f2a8c5bca60b1bb449c1b51"}
Sep 15 21:19:05.136904 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.136882    2551 generic.go:356] "Generic (PLEG): container finished" podID="0eb26d28-7baf-4cb6-b7f5-49941e1024e1" containerID="5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8" exitCode=0
Sep 15 21:19:05.136990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.136911    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b55bf787-jzlxh" event={"ID":"0eb26d28-7baf-4cb6-b7f5-49941e1024e1","Type":"ContainerDied","Data":"5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8"}
Sep 15 21:19:05.136990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.136918    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b55bf787-jzlxh"
Sep 15 21:19:05.136990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.136938    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b55bf787-jzlxh" event={"ID":"0eb26d28-7baf-4cb6-b7f5-49941e1024e1","Type":"ContainerDied","Data":"2e42e8e026edf119a93bcc65ccf16b0c6595e1a29e7b417d0a499f2430920712"}
Sep 15 21:19:05.136990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.136956    2551 scope.go:117] "RemoveContainer" containerID="5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8"
Sep 15 21:19:05.145431 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.145414    2551 scope.go:117] "RemoveContainer" containerID="5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8"
Sep 15 21:19:05.145679 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:19:05.145660    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8\": container with ID starting with 5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8 not found: ID does not exist" containerID="5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8"
Sep 15 21:19:05.145741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.145685    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8"} err="failed to get container status \"5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8\": rpc error: code = NotFound desc = could not find container \"5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8\": container with ID starting with 5bd7ff98860e930e8779b8361b19a4df8665b5111bf10e6878ee06145b09dee8 not found: ID does not exist"
Sep 15 21:19:05.150074 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150058    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.150127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150076    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tndt\" (UniqueName: \"kubernetes.io/projected/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-kube-api-access-8tndt\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.150127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150086    2551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-serving-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.150127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150094    2551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-oauth-config\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.150127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150102    2551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-oauth-serving-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.150127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150111    2551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-console-config\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.150127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.150119    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb26d28-7baf-4cb6-b7f5-49941e1024e1-trusted-ca-bundle\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:05.160342 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.160295    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" podStartSLOduration=2.16027849 podStartE2EDuration="2.16027849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:05.158903609 +0000 UTC m=+643.960772577" watchObservedRunningTime="2026-09-15 21:19:05.16027849 +0000 UTC m=+643.962147437"
Sep 15 21:19:05.173602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.173574    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74b55bf787-jzlxh"]
Sep 15 21:19:05.175061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.175041    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:05.178526 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.178499    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74b55bf787-jzlxh"]
Sep 15 21:19:05.180385 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.180369    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:05.769472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:05.769432    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb26d28-7baf-4cb6-b7f5-49941e1024e1" path="/var/lib/kubelet/pods/0eb26d28-7baf-4cb6-b7f5-49941e1024e1/volumes"
Sep 15 21:19:06.142783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.142696    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:06.143947 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.143928    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:19:06.254241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.254208    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:06.254548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.254536    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb26d28-7baf-4cb6-b7f5-49941e1024e1" containerName="console"
Sep 15 21:19:06.254593 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.254551    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb26d28-7baf-4cb6-b7f5-49941e1024e1" containerName="console"
Sep 15 21:19:06.254626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.254613    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb26d28-7baf-4cb6-b7f5-49941e1024e1" containerName="console"
Sep 15 21:19:06.257542 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.257523    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.259668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.259641    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 21:19:06.259779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.259697    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tf8sz\""
Sep 15 21:19:06.265762 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.265740    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:06.357439 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.357408    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:06.360937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.360912    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dkn\" (UniqueName: \"kubernetes.io/projected/a4622e50-e9ff-429a-8566-96371da8e833-kube-api-access-29dkn\") pod \"limitador-limitador-8588749977-l9g9x\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") " pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.361090 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.360950    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4622e50-e9ff-429a-8566-96371da8e833-config-file\") pod \"limitador-limitador-8588749977-l9g9x\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") " pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.462007 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.461968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dkn\" (UniqueName: \"kubernetes.io/projected/a4622e50-e9ff-429a-8566-96371da8e833-kube-api-access-29dkn\") pod \"limitador-limitador-8588749977-l9g9x\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") " pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.462213 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.462042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4622e50-e9ff-429a-8566-96371da8e833-config-file\") pod \"limitador-limitador-8588749977-l9g9x\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") " pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.467236 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.463602    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4622e50-e9ff-429a-8566-96371da8e833-config-file\") pod \"limitador-limitador-8588749977-l9g9x\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") " pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.475448 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.471957    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dkn\" (UniqueName: \"kubernetes.io/projected/a4622e50-e9ff-429a-8566-96371da8e833-kube-api-access-29dkn\") pod \"limitador-limitador-8588749977-l9g9x\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") " pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.568586 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.568551    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:06.840356 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.840272    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:06.843137 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:19:06.843102    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4622e50_e9ff_429a_8566_96371da8e833.slice/crio-cc55a84c02da1de2157c6201e3f212e68c59de5f2a7952ecd6152f3c5e370a6a WatchSource:0}: Error finding container cc55a84c02da1de2157c6201e3f212e68c59de5f2a7952ecd6152f3c5e370a6a: Status 404 returned error can't find the container with id cc55a84c02da1de2157c6201e3f212e68c59de5f2a7952ecd6152f3c5e370a6a
Sep 15 21:19:06.845262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:06.845247    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:19:07.117909 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.117806    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5svxt"]
Sep 15 21:19:07.120850 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.120829    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:07.123744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.123722    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-smst8\""
Sep 15 21:19:07.130152 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.130130    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5svxt"]
Sep 15 21:19:07.148697 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.148664    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x" event={"ID":"a4622e50-e9ff-429a-8566-96371da8e833","Type":"ContainerStarted","Data":"cc55a84c02da1de2157c6201e3f212e68c59de5f2a7952ecd6152f3c5e370a6a"}
Sep 15 21:19:07.171403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.171373    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7fv9\" (UniqueName: \"kubernetes.io/projected/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927-kube-api-access-f7fv9\") pod \"authorino-7bcfd8f9cd-5svxt\" (UID: \"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:07.272247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.272182    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7fv9\" (UniqueName: \"kubernetes.io/projected/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927-kube-api-access-f7fv9\") pod \"authorino-7bcfd8f9cd-5svxt\" (UID: \"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:07.283579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.283539    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7fv9\" (UniqueName: \"kubernetes.io/projected/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927-kube-api-access-f7fv9\") pod \"authorino-7bcfd8f9cd-5svxt\" (UID: \"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:07.430314 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.430278    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:07.700328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:07.700246    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5svxt"]
Sep 15 21:19:07.704003 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:19:07.703976    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81c6022a_4cde_45a5_a1e4_bbf6ea4fc927.slice/crio-e1e7244ef39c8ef8f1b453eaf38e34b615062a3b41711a5ee842b5ff4ed9e26b WatchSource:0}: Error finding container e1e7244ef39c8ef8f1b453eaf38e34b615062a3b41711a5ee842b5ff4ed9e26b: Status 404 returned error can't find the container with id e1e7244ef39c8ef8f1b453eaf38e34b615062a3b41711a5ee842b5ff4ed9e26b
Sep 15 21:19:08.154532 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:08.154496    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt" event={"ID":"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927","Type":"ContainerStarted","Data":"e1e7244ef39c8ef8f1b453eaf38e34b615062a3b41711a5ee842b5ff4ed9e26b"}
Sep 15 21:19:08.156438 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:08.156347    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x" event={"ID":"a4622e50-e9ff-429a-8566-96371da8e833","Type":"ContainerStarted","Data":"1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6"}
Sep 15 21:19:08.156633 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:08.156614    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:08.174573 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:08.174521    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x" podStartSLOduration=1.101490379 podStartE2EDuration="2.174505798s" totalImagesPullingTime="1.073015419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:08.173572085 +0000 UTC m=+646.975441038" watchObservedRunningTime="2026-09-15 21:19:08.174505798 +0000 UTC m=+646.976374741"
Sep 15 21:19:11.173397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:11.173356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt" event={"ID":"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927","Type":"ContainerStarted","Data":"0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c"}
Sep 15 21:19:11.193937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:11.193879    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt" podStartSLOduration=1.570589489 podStartE2EDuration="4.193861407s" totalImagesPullingTime="2.623271918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:11.192607087 +0000 UTC m=+649.994476029" watchObservedRunningTime="2026-09-15 21:19:11.193861407 +0000 UTC m=+649.995730350"
Sep 15 21:19:11.829926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:11.829882    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5svxt"]
Sep 15 21:19:13.183208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:13.183158    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt" podUID="81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" containerName="authorino" containerID="cri-o://0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c" gracePeriod=30
Sep 15 21:19:13.426604 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:13.426583    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:13.521859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:13.521764    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7fv9\" (UniqueName: \"kubernetes.io/projected/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927-kube-api-access-f7fv9\") pod \"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927\" (UID: \"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927\") "
Sep 15 21:19:13.523966 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:13.523930    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927-kube-api-access-f7fv9" (OuterVolumeSpecName: "kube-api-access-f7fv9") pod "81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" (UID: "81c6022a-4cde-45a5-a1e4-bbf6ea4fc927"). InnerVolumeSpecName "kube-api-access-f7fv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:13.622783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:13.622748    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7fv9\" (UniqueName: \"kubernetes.io/projected/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927-kube-api-access-f7fv9\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:14.188378 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.188343    2551 generic.go:356] "Generic (PLEG): container finished" podID="81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" containerID="0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c" exitCode=0
Sep 15 21:19:14.188813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.188386    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt"
Sep 15 21:19:14.188813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.188414    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt" event={"ID":"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927","Type":"ContainerDied","Data":"0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c"}
Sep 15 21:19:14.188813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.188455    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5svxt" event={"ID":"81c6022a-4cde-45a5-a1e4-bbf6ea4fc927","Type":"ContainerDied","Data":"e1e7244ef39c8ef8f1b453eaf38e34b615062a3b41711a5ee842b5ff4ed9e26b"}
Sep 15 21:19:14.188813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.188472    2551 scope.go:117] "RemoveContainer" containerID="0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c"
Sep 15 21:19:14.197142 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.197126    2551 scope.go:117] "RemoveContainer" containerID="0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c"
Sep 15 21:19:14.197494 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:19:14.197471    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c\": container with ID starting with 0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c not found: ID does not exist" containerID="0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c"
Sep 15 21:19:14.197548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.197509    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c"} err="failed to get container status \"0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c\": rpc error: code = NotFound desc = could not find container \"0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c\": container with ID starting with 0aa0ab63f45483af22db0bac4b25be6b4f164ebad4275725cf5f6f51cf72e89c not found: ID does not exist"
Sep 15 21:19:14.206691 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.206660    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5svxt"]
Sep 15 21:19:14.212537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:14.212510    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5svxt"]
Sep 15 21:19:15.767600 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:15.767568    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" path="/var/lib/kubelet/pods/81c6022a-4cde-45a5-a1e4-bbf6ea4fc927/volumes"
Sep 15 21:19:19.162979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:19.162944    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:23.833343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:23.833307    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:23.833803 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:23.833502    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x" podUID="a4622e50-e9ff-429a-8566-96371da8e833" containerName="limitador" containerID="cri-o://1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6" gracePeriod=30
Sep 15 21:19:24.375037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.375014    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:24.522316 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.522286    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4622e50-e9ff-429a-8566-96371da8e833-config-file\") pod \"a4622e50-e9ff-429a-8566-96371da8e833\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") "
Sep 15 21:19:24.522496 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.522323    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29dkn\" (UniqueName: \"kubernetes.io/projected/a4622e50-e9ff-429a-8566-96371da8e833-kube-api-access-29dkn\") pod \"a4622e50-e9ff-429a-8566-96371da8e833\" (UID: \"a4622e50-e9ff-429a-8566-96371da8e833\") "
Sep 15 21:19:24.522769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.522730    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4622e50-e9ff-429a-8566-96371da8e833-config-file" (OuterVolumeSpecName: "config-file") pod "a4622e50-e9ff-429a-8566-96371da8e833" (UID: "a4622e50-e9ff-429a-8566-96371da8e833"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:19:24.524601 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.524575    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4622e50-e9ff-429a-8566-96371da8e833-kube-api-access-29dkn" (OuterVolumeSpecName: "kube-api-access-29dkn") pod "a4622e50-e9ff-429a-8566-96371da8e833" (UID: "a4622e50-e9ff-429a-8566-96371da8e833"). InnerVolumeSpecName "kube-api-access-29dkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:24.623678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.623639    2551 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4622e50-e9ff-429a-8566-96371da8e833-config-file\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:24.623678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:24.623678    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29dkn\" (UniqueName: \"kubernetes.io/projected/a4622e50-e9ff-429a-8566-96371da8e833-kube-api-access-29dkn\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:25.236387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.236348    2551 generic.go:356] "Generic (PLEG): container finished" podID="a4622e50-e9ff-429a-8566-96371da8e833" containerID="1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6" exitCode=0
Sep 15 21:19:25.236853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.236394    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x"
Sep 15 21:19:25.236853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.236416    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x" event={"ID":"a4622e50-e9ff-429a-8566-96371da8e833","Type":"ContainerDied","Data":"1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6"}
Sep 15 21:19:25.236853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.236470    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l9g9x" event={"ID":"a4622e50-e9ff-429a-8566-96371da8e833","Type":"ContainerDied","Data":"cc55a84c02da1de2157c6201e3f212e68c59de5f2a7952ecd6152f3c5e370a6a"}
Sep 15 21:19:25.236853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.236503    2551 scope.go:117] "RemoveContainer" containerID="1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6"
Sep 15 21:19:25.245165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.245151    2551 scope.go:117] "RemoveContainer" containerID="1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6"
Sep 15 21:19:25.245494 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:19:25.245474    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6\": container with ID starting with 1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6 not found: ID does not exist" containerID="1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6"
Sep 15 21:19:25.245548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.245503    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6"} err="failed to get container status \"1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6\": rpc error: code = NotFound desc = could not find container \"1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6\": container with ID starting with 1e91149f9a474d59df6ee050ce858d8e89c81b3d3c1f73e3302734894d2689d6 not found: ID does not exist"
Sep 15 21:19:25.258938 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.258908    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:25.263520 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.263498    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l9g9x"]
Sep 15 21:19:25.767819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:25.767788    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4622e50-e9ff-429a-8566-96371da8e833" path="/var/lib/kubelet/pods/a4622e50-e9ff-429a-8566-96371da8e833/volumes"
Sep 15 21:19:27.791782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.791749    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9dths"]
Sep 15 21:19:27.792299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.792058    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4622e50-e9ff-429a-8566-96371da8e833" containerName="limitador"
Sep 15 21:19:27.792299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.792074    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4622e50-e9ff-429a-8566-96371da8e833" containerName="limitador"
Sep 15 21:19:27.792299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.792101    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" containerName="authorino"
Sep 15 21:19:27.792299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.792106    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" containerName="authorino"
Sep 15 21:19:27.792299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.792182    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c6022a-4cde-45a5-a1e4-bbf6ea4fc927" containerName="authorino"
Sep 15 21:19:27.792299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.792208    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4622e50-e9ff-429a-8566-96371da8e833" containerName="limitador"
Sep 15 21:19:27.795272 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.795251    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:27.798062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.797960    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:19:27.798062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.797995    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 15 21:19:27.798062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.798055    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wd87j\""
Sep 15 21:19:27.798779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.798761    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:19:27.806093 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.806070    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9dths"]
Sep 15 21:19:27.955020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.954986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/362d8645-020f-4cc0-b9d5-1f7376c6457e-data\") pod \"postgres-69fb4594cf-9dths\" (UID: \"362d8645-020f-4cc0-b9d5-1f7376c6457e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:27.955173 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:27.955037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bm2\" (UniqueName: \"kubernetes.io/projected/362d8645-020f-4cc0-b9d5-1f7376c6457e-kube-api-access-d4bm2\") pod \"postgres-69fb4594cf-9dths\" (UID: \"362d8645-020f-4cc0-b9d5-1f7376c6457e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:28.056256 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:28.056137    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/362d8645-020f-4cc0-b9d5-1f7376c6457e-data\") pod \"postgres-69fb4594cf-9dths\" (UID: \"362d8645-020f-4cc0-b9d5-1f7376c6457e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:28.056256 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:28.056220    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bm2\" (UniqueName: \"kubernetes.io/projected/362d8645-020f-4cc0-b9d5-1f7376c6457e-kube-api-access-d4bm2\") pod \"postgres-69fb4594cf-9dths\" (UID: \"362d8645-020f-4cc0-b9d5-1f7376c6457e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:28.056527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:28.056500    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/362d8645-020f-4cc0-b9d5-1f7376c6457e-data\") pod \"postgres-69fb4594cf-9dths\" (UID: \"362d8645-020f-4cc0-b9d5-1f7376c6457e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:28.072508 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:28.072477    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bm2\" (UniqueName: \"kubernetes.io/projected/362d8645-020f-4cc0-b9d5-1f7376c6457e-kube-api-access-d4bm2\") pod \"postgres-69fb4594cf-9dths\" (UID: \"362d8645-020f-4cc0-b9d5-1f7376c6457e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:28.106503 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:28.106467    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:28.375744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:28.375720    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9dths"]
Sep 15 21:19:28.378104 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:19:28.378074    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362d8645_020f_4cc0_b9d5_1f7376c6457e.slice/crio-a96ce61a064678f776d9ab35b6a52ccacff9d1bcd360f0e22c6c6187bbea02d6 WatchSource:0}: Error finding container a96ce61a064678f776d9ab35b6a52ccacff9d1bcd360f0e22c6c6187bbea02d6: Status 404 returned error can't find the container with id a96ce61a064678f776d9ab35b6a52ccacff9d1bcd360f0e22c6c6187bbea02d6
Sep 15 21:19:29.258069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:29.257980    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths" event={"ID":"362d8645-020f-4cc0-b9d5-1f7376c6457e","Type":"ContainerStarted","Data":"a96ce61a064678f776d9ab35b6a52ccacff9d1bcd360f0e22c6c6187bbea02d6"}
Sep 15 21:19:33.280256 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:33.280224    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths" event={"ID":"362d8645-020f-4cc0-b9d5-1f7376c6457e","Type":"ContainerStarted","Data":"6addf48fba5b77bd2f8a9286b320eb0617366c8d790d0462a18ffc4d5eb52bd0"}
Sep 15 21:19:33.280640 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:33.280342    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:33.297502 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:33.297440    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths" podStartSLOduration=1.595732777 podStartE2EDuration="6.297418761s" totalImagesPullingTime="4.701685984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:33.295180932 +0000 UTC m=+672.097049904" watchObservedRunningTime="2026-09-15 21:19:33.297418761 +0000 UTC m=+672.099287705"
Sep 15 21:19:39.315160 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.315129    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9dths"
Sep 15 21:19:39.853312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.853278    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-lr6rm"]
Sep 15 21:19:39.855698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.855675    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:39.858128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.858110    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-smst8\""
Sep 15 21:19:39.864062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.864040    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-lr6rm"]
Sep 15 21:19:39.958440 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.958410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4j26\" (UniqueName: \"kubernetes.io/projected/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4-kube-api-access-x4j26\") pod \"authorino-557f876f69-lr6rm\" (UID: \"84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4\") " pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:39.969629 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:39.969603    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lr6rm"]
Sep 15 21:19:39.969833 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:19:39.969815    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-x4j26], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-lr6rm" podUID="84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4"
Sep 15 21:19:40.001819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.001795    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gq8kx"]
Sep 15 21:19:40.004349 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.004332    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.006540 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.006513    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 21:19:40.013434 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.013410    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gq8kx"]
Sep 15 21:19:40.059122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.059087    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwg6x\" (UniqueName: \"kubernetes.io/projected/02f95df4-a78e-4c72-881d-d43b5caf48c3-kube-api-access-vwg6x\") pod \"authorino-6d4dfc649f-gq8kx\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") " pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.059122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.059129    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4j26\" (UniqueName: \"kubernetes.io/projected/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4-kube-api-access-x4j26\") pod \"authorino-557f876f69-lr6rm\" (UID: \"84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4\") " pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:40.059366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.059144    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02f95df4-a78e-4c72-881d-d43b5caf48c3-tls-cert\") pod \"authorino-6d4dfc649f-gq8kx\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") " pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.072088 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.072058    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4j26\" (UniqueName: \"kubernetes.io/projected/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4-kube-api-access-x4j26\") pod \"authorino-557f876f69-lr6rm\" (UID: \"84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4\") " pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:40.085501 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.085466    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gq8kx"]
Sep 15 21:19:40.085751 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:19:40.085727    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vwg6x tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-gq8kx" podUID="02f95df4-a78e-4c72-881d-d43b5caf48c3"
Sep 15 21:19:40.160297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.160269    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwg6x\" (UniqueName: \"kubernetes.io/projected/02f95df4-a78e-4c72-881d-d43b5caf48c3-kube-api-access-vwg6x\") pod \"authorino-6d4dfc649f-gq8kx\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") " pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.160441 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.160311    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02f95df4-a78e-4c72-881d-d43b5caf48c3-tls-cert\") pod \"authorino-6d4dfc649f-gq8kx\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") " pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.162868 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.162835    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02f95df4-a78e-4c72-881d-d43b5caf48c3-tls-cert\") pod \"authorino-6d4dfc649f-gq8kx\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") " pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.168244 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.168221    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwg6x\" (UniqueName: \"kubernetes.io/projected/02f95df4-a78e-4c72-881d-d43b5caf48c3-kube-api-access-vwg6x\") pod \"authorino-6d4dfc649f-gq8kx\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") " pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.311594 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.311554    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.312030 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.312012    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:40.325054 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.325031    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:40.326320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.326299    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:40.362310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.362284    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4j26\" (UniqueName: \"kubernetes.io/projected/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4-kube-api-access-x4j26\") pod \"84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4\" (UID: \"84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4\") "
Sep 15 21:19:40.362484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.362358    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwg6x\" (UniqueName: \"kubernetes.io/projected/02f95df4-a78e-4c72-881d-d43b5caf48c3-kube-api-access-vwg6x\") pod \"02f95df4-a78e-4c72-881d-d43b5caf48c3\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") "
Sep 15 21:19:40.362484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.362406    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02f95df4-a78e-4c72-881d-d43b5caf48c3-tls-cert\") pod \"02f95df4-a78e-4c72-881d-d43b5caf48c3\" (UID: \"02f95df4-a78e-4c72-881d-d43b5caf48c3\") "
Sep 15 21:19:40.364542 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.364514    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4-kube-api-access-x4j26" (OuterVolumeSpecName: "kube-api-access-x4j26") pod "84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4" (UID: "84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4"). InnerVolumeSpecName "kube-api-access-x4j26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:40.364646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.364598    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f95df4-a78e-4c72-881d-d43b5caf48c3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "02f95df4-a78e-4c72-881d-d43b5caf48c3" (UID: "02f95df4-a78e-4c72-881d-d43b5caf48c3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:19:40.364646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.364605    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f95df4-a78e-4c72-881d-d43b5caf48c3-kube-api-access-vwg6x" (OuterVolumeSpecName: "kube-api-access-vwg6x") pod "02f95df4-a78e-4c72-881d-d43b5caf48c3" (UID: "02f95df4-a78e-4c72-881d-d43b5caf48c3"). InnerVolumeSpecName "kube-api-access-vwg6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:19:40.463261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.463156    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwg6x\" (UniqueName: \"kubernetes.io/projected/02f95df4-a78e-4c72-881d-d43b5caf48c3-kube-api-access-vwg6x\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.463261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.463224    2551 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02f95df4-a78e-4c72-881d-d43b5caf48c3-tls-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:40.463261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:40.463235    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4j26\" (UniqueName: \"kubernetes.io/projected/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4-kube-api-access-x4j26\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:19:41.315933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.315901    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-gq8kx"
Sep 15 21:19:41.316108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.315901    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lr6rm"
Sep 15 21:19:41.350808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.350773    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lr6rm"]
Sep 15 21:19:41.352400 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.352374    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-lr6rm"]
Sep 15 21:19:41.372226 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.372178    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gq8kx"]
Sep 15 21:19:41.377673 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.377649    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-gq8kx"]
Sep 15 21:19:41.768209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.768147    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f95df4-a78e-4c72-881d-d43b5caf48c3" path="/var/lib/kubelet/pods/02f95df4-a78e-4c72-881d-d43b5caf48c3/volumes"
Sep 15 21:19:41.768417 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:41.768404    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4" path="/var/lib/kubelet/pods/84f2bbb2-e0d0-458a-8d1c-8f971f45bcc4/volumes"
Sep 15 21:19:46.682578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.682546    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6865b5c455-px559"]
Sep 15 21:19:46.684970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.684954    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.687386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.687361    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:19:46.687518 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.687368    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:19:46.687518 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.687363    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:19:46.687518 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.687371    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wvswp\""
Sep 15 21:19:46.699274 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.699246    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6865b5c455-px559"]
Sep 15 21:19:46.718796 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.718768    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7sm\" (UniqueName: \"kubernetes.io/projected/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-kube-api-access-kp7sm\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.718950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.718808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-webhook-cert\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.718950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.718909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-metrics-certs\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.820345 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.820307    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7sm\" (UniqueName: \"kubernetes.io/projected/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-kube-api-access-kp7sm\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.820345 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.820349    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-webhook-cert\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.820549 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.820395    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-metrics-certs\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.822968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.822936    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-metrics-certs\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.823085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.822978    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-webhook-cert\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.827948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.827925    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7sm\" (UniqueName: \"kubernetes.io/projected/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-kube-api-access-kp7sm\") pod \"maas-controller-6865b5c455-px559\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") " pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:46.994867 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:46.994786    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:47.300284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:47.300256    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6865b5c455-px559"]
Sep 15 21:19:47.300647 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:19:47.300626    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5276c2a_a0f1_40ba_be1f_b6c0166426d0.slice/crio-cd298e2dbdb227320353e1fcc7c17564e3cdc7d1effa3c198c0e1e72f2ad06ec WatchSource:0}: Error finding container cd298e2dbdb227320353e1fcc7c17564e3cdc7d1effa3c198c0e1e72f2ad06ec: Status 404 returned error can't find the container with id cd298e2dbdb227320353e1fcc7c17564e3cdc7d1effa3c198c0e1e72f2ad06ec
Sep 15 21:19:47.340993 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:47.340957    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865b5c455-px559" event={"ID":"b5276c2a-a0f1-40ba-be1f-b6c0166426d0","Type":"ContainerStarted","Data":"cd298e2dbdb227320353e1fcc7c17564e3cdc7d1effa3c198c0e1e72f2ad06ec"}
Sep 15 21:19:50.358582 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:50.358544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865b5c455-px559" event={"ID":"b5276c2a-a0f1-40ba-be1f-b6c0166426d0","Type":"ContainerStarted","Data":"ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3"}
Sep 15 21:19:50.359058 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:50.358645    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:19:50.376832 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:19:50.376785    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6865b5c455-px559" podStartSLOduration=2.150689043 podStartE2EDuration="4.37676916s" totalImagesPullingTime="2.226080117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:19:50.374731472 +0000 UTC m=+689.176600414" watchObservedRunningTime="2026-09-15 21:19:50.37676916 +0000 UTC m=+689.178638102"
Sep 15 21:20:00.856686 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:00.856651    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6865b5c455-px559"]
Sep 15 21:20:00.857101 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:00.856853    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6865b5c455-px559" podUID="b5276c2a-a0f1-40ba-be1f-b6c0166426d0" containerName="manager" containerID="cri-o://ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3" gracePeriod=30
Sep 15 21:20:00.858560 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:00.858538    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:20:01.096625 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.096603    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:20:01.146138 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.146056    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-webhook-cert\") pod \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") "
Sep 15 21:20:01.146138 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.146104    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp7sm\" (UniqueName: \"kubernetes.io/projected/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-kube-api-access-kp7sm\") pod \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") "
Sep 15 21:20:01.146378 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.146141    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-metrics-certs\") pod \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\" (UID: \"b5276c2a-a0f1-40ba-be1f-b6c0166426d0\") "
Sep 15 21:20:01.148467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.148429    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b5276c2a-a0f1-40ba-be1f-b6c0166426d0" (UID: "b5276c2a-a0f1-40ba-be1f-b6c0166426d0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:20:01.148578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.148473    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-kube-api-access-kp7sm" (OuterVolumeSpecName: "kube-api-access-kp7sm") pod "b5276c2a-a0f1-40ba-be1f-b6c0166426d0" (UID: "b5276c2a-a0f1-40ba-be1f-b6c0166426d0"). InnerVolumeSpecName "kube-api-access-kp7sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:20:01.148578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.148486    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b5276c2a-a0f1-40ba-be1f-b6c0166426d0" (UID: "b5276c2a-a0f1-40ba-be1f-b6c0166426d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:20:01.247390 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.247354    2551 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-webhook-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:20:01.247390 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.247384    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7sm\" (UniqueName: \"kubernetes.io/projected/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-kube-api-access-kp7sm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:20:01.247390 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.247394    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5276c2a-a0f1-40ba-be1f-b6c0166426d0-metrics-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:20:01.406048 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.405966    2551 generic.go:356] "Generic (PLEG): container finished" podID="b5276c2a-a0f1-40ba-be1f-b6c0166426d0" containerID="ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3" exitCode=0
Sep 15 21:20:01.406048 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.406007    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6865b5c455-px559"
Sep 15 21:20:01.406218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.406054    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865b5c455-px559" event={"ID":"b5276c2a-a0f1-40ba-be1f-b6c0166426d0","Type":"ContainerDied","Data":"ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3"}
Sep 15 21:20:01.406218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.406096    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6865b5c455-px559" event={"ID":"b5276c2a-a0f1-40ba-be1f-b6c0166426d0","Type":"ContainerDied","Data":"cd298e2dbdb227320353e1fcc7c17564e3cdc7d1effa3c198c0e1e72f2ad06ec"}
Sep 15 21:20:01.406218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.406117    2551 scope.go:117] "RemoveContainer" containerID="ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3"
Sep 15 21:20:01.414716 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.414697    2551 scope.go:117] "RemoveContainer" containerID="ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3"
Sep 15 21:20:01.415011 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:20:01.414990    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3\": container with ID starting with ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3 not found: ID does not exist" containerID="ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3"
Sep 15 21:20:01.415077 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.415019    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3"} err="failed to get container status \"ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3\": rpc error: code = NotFound desc = could not find container \"ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3\": container with ID starting with ca6c2d8f338e62a66981796147d7236e8c870038d74e39184688290e11c463a3 not found: ID does not exist"
Sep 15 21:20:01.426660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.426636    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6865b5c455-px559"]
Sep 15 21:20:01.430754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.430730    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6865b5c455-px559"]
Sep 15 21:20:01.767678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:01.767639    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5276c2a-a0f1-40ba-be1f-b6c0166426d0" path="/var/lib/kubelet/pods/b5276c2a-a0f1-40ba-be1f-b6c0166426d0/volumes"
Sep 15 21:20:23.003976 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.003945    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"]
Sep 15 21:20:23.004370 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.004279    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5276c2a-a0f1-40ba-be1f-b6c0166426d0" containerName="manager"
Sep 15 21:20:23.004370 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.004291    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5276c2a-a0f1-40ba-be1f-b6c0166426d0" containerName="manager"
Sep 15 21:20:23.004370 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.004350    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5276c2a-a0f1-40ba-be1f-b6c0166426d0" containerName="manager"
Sep 15 21:20:23.008397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.008380    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.010805 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.010787    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vzmv2\""
Sep 15 21:20:23.012647 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.012627    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 21:20:23.013100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.013083    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:20:23.023002 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.022976    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"]
Sep 15 21:20:23.042451 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.042425    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.042581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.042462    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.042581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.042562    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg8kw\" (UniqueName: \"kubernetes.io/projected/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-kube-api-access-rg8kw\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.042649 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.042626    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-openshift-service-ca\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.142960 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.142929    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg8kw\" (UniqueName: \"kubernetes.io/projected/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-kube-api-access-rg8kw\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.143122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.142972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-openshift-service-ca\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.143122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.142995    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.143122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.143016    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.143122 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:20:23.143086    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 15 21:20:23.143122 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:20:23.143106    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 15 21:20:23.143331 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:20:23.143164    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls podName:7dd7a051-941c-42ef-b1fa-cb4915f75b5f nodeName:}" failed. No retries permitted until 2026-09-15 21:20:23.643144366 +0000 UTC m=+722.445013294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls") pod "maas-api-74c7fc967-x7vnq" (UID: "7dd7a051-941c-42ef-b1fa-cb4915f75b5f") : secret "maas-api-serving-cert" not found
Sep 15 21:20:23.143331 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:20:23.143183    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs podName:7dd7a051-941c-42ef-b1fa-cb4915f75b5f nodeName:}" failed. No retries permitted until 2026-09-15 21:20:23.643174272 +0000 UTC m=+722.445043194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs") pod "maas-api-74c7fc967-x7vnq" (UID: "7dd7a051-941c-42ef-b1fa-cb4915f75b5f") : secret "maas-api-metrics-tls" not found
Sep 15 21:20:23.143783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.143763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-openshift-service-ca\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.152001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.151977    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg8kw\" (UniqueName: \"kubernetes.io/projected/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-kube-api-access-rg8kw\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.645781 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.645748    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.645781 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.645786    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.648404 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.648377    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.648546 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.648401    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs\") pod \"maas-api-74c7fc967-x7vnq\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") " pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:23.917959 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:23.917928    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:24.204653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:24.204569    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"]
Sep 15 21:20:24.208440 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:20:24.208411    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd7a051_941c_42ef_b1fa_cb4915f75b5f.slice/crio-094a3a5f6a333d59bb431c8a18825fba01bf5e6cf10cfd9639a1e22e0f3ea2bf WatchSource:0}: Error finding container 094a3a5f6a333d59bb431c8a18825fba01bf5e6cf10cfd9639a1e22e0f3ea2bf: Status 404 returned error can't find the container with id 094a3a5f6a333d59bb431c8a18825fba01bf5e6cf10cfd9639a1e22e0f3ea2bf
Sep 15 21:20:24.515953 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:24.515863    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" event={"ID":"7dd7a051-941c-42ef-b1fa-cb4915f75b5f","Type":"ContainerStarted","Data":"094a3a5f6a333d59bb431c8a18825fba01bf5e6cf10cfd9639a1e22e0f3ea2bf"}
Sep 15 21:20:27.535220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:27.535125    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" event={"ID":"7dd7a051-941c-42ef-b1fa-cb4915f75b5f","Type":"ContainerStarted","Data":"f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9"}
Sep 15 21:20:27.535220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:27.535212    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:27.552786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:27.552732    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" podStartSLOduration=2.597686367 podStartE2EDuration="5.552715517s" totalImagesPullingTime="2.95502915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:27.552176332 +0000 UTC m=+726.354045275" watchObservedRunningTime="2026-09-15 21:20:27.552715517 +0000 UTC m=+726.354584459"
Sep 15 21:20:32.559463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:32.559379    2551 generic.go:356] "Generic (PLEG): container finished" podID="7aee3b2d-7f62-48a5-89cf-e0d93904115f" containerID="c187a4b1c5c15407b26bfd4877e3b6b28c8b6ce45c6800da17476237905ba03c" exitCode=0
Sep 15 21:20:32.559463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:32.559436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" event={"ID":"7aee3b2d-7f62-48a5-89cf-e0d93904115f","Type":"ContainerDied","Data":"c187a4b1c5c15407b26bfd4877e3b6b28c8b6ce45c6800da17476237905ba03c"}
Sep 15 21:20:32.559910 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:32.559782    2551 scope.go:117] "RemoveContainer" containerID="c187a4b1c5c15407b26bfd4877e3b6b28c8b6ce45c6800da17476237905ba03c"
Sep 15 21:20:32.560785 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:32.560753    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:20:32.560862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:32.560844    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:20:32.560915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:32.560898    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:20:33.545419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:33.545392    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:20:33.566164 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:33.566129    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" event={"ID":"7aee3b2d-7f62-48a5-89cf-e0d93904115f","Type":"ContainerStarted","Data":"e6a8406d67b6f45321ad9b76fbdee8abce4f14e0e7f3be154493cb4da89b6c47"}
Sep 15 21:20:34.175581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:34.175538    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:20:34.175793 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:34.175692    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8xm2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 15 21:20:34.175793 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:34.175737    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" podUID="7aee3b2d-7f62-48a5-89cf-e0d93904115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 15 21:20:35.174728 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:35.174687    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8xm2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 15 21:20:35.175369 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:35.175336    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" podUID="7aee3b2d-7f62-48a5-89cf-e0d93904115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 15 21:20:36.143319 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:36.143286    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:20:36.175079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:36.175035    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8xm2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 15 21:20:36.175574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:36.175102    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" podUID="7aee3b2d-7f62-48a5-89cf-e0d93904115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 15 21:20:37.175114 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:37.175068    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8xm2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 15 21:20:37.175647 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:37.175137    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" podUID="7aee3b2d-7f62-48a5-89cf-e0d93904115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 15 21:20:38.174489 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:38.174449    2551 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-8xm2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Sep 15 21:20:38.174780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:38.174750    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l" podUID="7aee3b2d-7f62-48a5-89cf-e0d93904115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Sep 15 21:20:39.178493 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:39.178463    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:20:39.179386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:39.179367    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-8xm2l"
Sep 15 21:20:49.235434 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.235401    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"]
Sep 15 21:20:49.237922 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.237902    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.240465 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.240443    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:20:49.241380 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.241359    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 21:20:49.241490 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.241375    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:20:49.241490 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.241432    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2j7d9\""
Sep 15 21:20:49.249068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.249045    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"]
Sep 15 21:20:49.394592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.394561    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.394746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.394624    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gg7\" (UniqueName: \"kubernetes.io/projected/15ae6ce1-b938-4232-90ac-e0d8118f4666-kube-api-access-v7gg7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.394746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.394648    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.394746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.394702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.394746 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.394737    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.394883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.394753    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae6ce1-b938-4232-90ac-e0d8118f4666-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.495773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.495684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.495773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.495733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.495773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.495749    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae6ce1-b938-4232-90ac-e0d8118f4666-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.495773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.495771    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.496083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.495816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gg7\" (UniqueName: \"kubernetes.io/projected/15ae6ce1-b938-4232-90ac-e0d8118f4666-kube-api-access-v7gg7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.496083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.495837    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.496221 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.496149    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.496349 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.496330    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.496430 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.496407    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.498243 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.498213    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae6ce1-b938-4232-90ac-e0d8118f4666-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.498553 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.498534    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae6ce1-b938-4232-90ac-e0d8118f4666-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.504475 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.504451    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gg7\" (UniqueName: \"kubernetes.io/projected/15ae6ce1-b938-4232-90ac-e0d8118f4666-kube-api-access-v7gg7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd\" (UID: \"15ae6ce1-b938-4232-90ac-e0d8118f4666\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.547883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.547854    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:49.821010 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:49.820978    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"]
Sep 15 21:20:49.823410 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:20:49.823383    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ae6ce1_b938_4232_90ac_e0d8118f4666.slice/crio-8135d517a7a49edcc29d3635cc54064b1c109fb9c43b1e444f2476aec68b52e6 WatchSource:0}: Error finding container 8135d517a7a49edcc29d3635cc54064b1c109fb9c43b1e444f2476aec68b52e6: Status 404 returned error can't find the container with id 8135d517a7a49edcc29d3635cc54064b1c109fb9c43b1e444f2476aec68b52e6
Sep 15 21:20:50.649892 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:50.649858    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd" event={"ID":"15ae6ce1-b938-4232-90ac-e0d8118f4666","Type":"ContainerStarted","Data":"8135d517a7a49edcc29d3635cc54064b1c109fb9c43b1e444f2476aec68b52e6"}
Sep 15 21:20:51.656065 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:51.656031    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd" event={"ID":"15ae6ce1-b938-4232-90ac-e0d8118f4666","Type":"ContainerStarted","Data":"9fbd03b6a17bd0b856468b501ecd03b42c55518ad74e28fe7ae6a5d7cd2d4adf"}
Sep 15 21:20:51.672577 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:51.672273    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd" podStartSLOduration=1.5063415039999999 podStartE2EDuration="2.672252421s" totalImagesPullingTime="1.165910917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:20:51.672057938 +0000 UTC m=+750.473926918" watchObservedRunningTime="2026-09-15 21:20:51.672252421 +0000 UTC m=+750.474121368"
Sep 15 21:20:59.548498 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:59.548456    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:59.549139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:59.548528    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:59.561129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:59.561102    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:20:59.700660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:20:59.700629    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd"
Sep 15 21:21:24.234004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.233972    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"]
Sep 15 21:21:24.237280 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.237259    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.240018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.239992    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 21:21:24.251536 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.251511    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"]
Sep 15 21:21:24.278354 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.278324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.278500 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.278411    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s84l\" (UniqueName: \"kubernetes.io/projected/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-kube-api-access-5s84l\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.278500 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.278443    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.278500 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.278461    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.278500 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.278486    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.278634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.278518    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.379750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.379717    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.379750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.379755    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.379977 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.379771    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.379977 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.379810    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.379977 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.379904    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s84l\" (UniqueName: \"kubernetes.io/projected/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-kube-api-access-5s84l\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.379977 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.379948    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.380220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.380166    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.380354 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.380331    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.380417 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.380337    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.382469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.382446    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.382630 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.382613    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.388629 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.388607    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s84l\" (UniqueName: \"kubernetes.io/projected/aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2-kube-api-access-5s84l\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr\" (UID: \"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.547501 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.547416    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:24.821091 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:24.821068    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"]
Sep 15 21:21:24.822940 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:21:24.822916    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb2fe73_2ac3_4a9e_8600_7998fa8fe1f2.slice/crio-e39524b3aa9b5aea5cfd7a3ac8fbc36d916b49c5ff0b395d2c224cd1420041b4 WatchSource:0}: Error finding container e39524b3aa9b5aea5cfd7a3ac8fbc36d916b49c5ff0b395d2c224cd1420041b4: Status 404 returned error can't find the container with id e39524b3aa9b5aea5cfd7a3ac8fbc36d916b49c5ff0b395d2c224cd1420041b4
Sep 15 21:21:25.802789 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:25.802748    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr" event={"ID":"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2","Type":"ContainerStarted","Data":"fca951ba40a16b3e03d91f28cd02c72cd24c00ee367bcc6023f13d387115bf4c"}
Sep 15 21:21:25.803239 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:25.802816    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr" event={"ID":"aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2","Type":"ContainerStarted","Data":"e39524b3aa9b5aea5cfd7a3ac8fbc36d916b49c5ff0b395d2c224cd1420041b4"}
Sep 15 21:21:25.830430 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:25.824394    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr" podStartSLOduration=1.5975147939999998 podStartE2EDuration="1.824371607s" totalImagesPullingTime="226.856813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:25.820778446 +0000 UTC m=+784.622647409" watchObservedRunningTime="2026-09-15 21:21:25.824371607 +0000 UTC m=+784.626240551"
Sep 15 21:21:34.547862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:34.547823    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:34.548363 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:34.547904    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:34.560323 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:34.560296    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:34.852226 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:34.852128    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr"
Sep 15 21:21:49.833151 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:49.833064    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"]
Sep 15 21:21:49.836448 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:49.836426    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:49.839141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:49.839120    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 21:21:49.848641 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:49.848617    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"]
Sep 15 21:21:50.001797 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.001770    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.001951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.001808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.001951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.001834    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.001951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.001850    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.001951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.001911    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.002086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.001955    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhwn\" (UniqueName: \"kubernetes.io/projected/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-kube-api-access-mwhwn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103514 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103430    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103514 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103480    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103603    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103636    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103658    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103702    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhwn\" (UniqueName: \"kubernetes.io/projected/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-kube-api-access-mwhwn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103918 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103859    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.103978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.103922    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.104038 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.104005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.106035 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.106008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.106221 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.106184    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.113431 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.113407    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhwn\" (UniqueName: \"kubernetes.io/projected/cc09a101-fc96-4a0e-8777-88d9a61f3bf9-kube-api-access-mwhwn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb\" (UID: \"cc09a101-fc96-4a0e-8777-88d9a61f3bf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.147423 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.147386    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:21:50.419959 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.419854    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"]
Sep 15 21:21:50.422834 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:21:50.422804    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc09a101_fc96_4a0e_8777_88d9a61f3bf9.slice/crio-89d6f05576413370a7e9f50cc187a3b20eb6b70326e058648314e768769117a7 WatchSource:0}: Error finding container 89d6f05576413370a7e9f50cc187a3b20eb6b70326e058648314e768769117a7: Status 404 returned error can't find the container with id 89d6f05576413370a7e9f50cc187a3b20eb6b70326e058648314e768769117a7
Sep 15 21:21:50.918774 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.918733    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb" event={"ID":"cc09a101-fc96-4a0e-8777-88d9a61f3bf9","Type":"ContainerStarted","Data":"a0121a02bd6230c5d4010cffa79412a9fa7d301280d75fda4f4e77552e2b080f"}
Sep 15 21:21:50.919244 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.918828    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb" event={"ID":"cc09a101-fc96-4a0e-8777-88d9a61f3bf9","Type":"ContainerStarted","Data":"89d6f05576413370a7e9f50cc187a3b20eb6b70326e058648314e768769117a7"}
Sep 15 21:21:50.938286 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:21:50.938236    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb" podStartSLOduration=1.755056629 podStartE2EDuration="1.938220826s" totalImagesPullingTime="183.164197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:50.936800844 +0000 UTC m=+809.738669811" watchObservedRunningTime="2026-09-15 21:21:50.938220826 +0000 UTC m=+809.740089760"
Sep 15 21:22:00.147569 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:22:00.147530    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:22:00.148058 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:22:00.147662    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:22:00.160341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:22:00.160313    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:22:00.980999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:22:00.980972    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb"
Sep 15 21:23:55.287330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.287299    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"]
Sep 15 21:23:55.290732 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.290714    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.293761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.293740    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-70880c-openshift-default-dockercfg-4fxhb\""
Sep 15 21:23:55.312530 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.312503    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"]
Sep 15 21:23:55.387111 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387074    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-certs\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-podinfo\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387221    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-envoy\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-socket\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387327    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-credential-socket\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387365    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-token\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387431    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-data\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387451    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrpt\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-kube-api-access-jjrpt\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.387478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.387469    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istiod-ca-cert\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488252 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488208    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-envoy\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488269    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-socket\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488313    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-credential-socket\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488336    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-token\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488382    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-data\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488612 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488410    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrpt\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-kube-api-access-jjrpt\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488612 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488439    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istiod-ca-cert\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488612 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-certs\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488612 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488512    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-podinfo\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488701    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-socket\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488762    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-data\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488923    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-certs\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.488979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.488970    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-credential-socket\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.489160 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.489142    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istiod-ca-cert\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.490707 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.490682    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-envoy\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.491180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.491165    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-podinfo\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.499510 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.499484    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"]
Sep 15 21:23:55.503047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.503032    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.506616 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.506595    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a276f9-openshift-default-dockercfg-knl7h\""
Sep 15 21:23:55.530838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.530812    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"]
Sep 15 21:23:55.531431 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.531412    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrpt\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-kube-api-access-jjrpt\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.534720 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.534699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-token\") pod \"e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") " pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.589714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589627    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-podinfo\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-token\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589690    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-credential-socket\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-certs\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-envoy\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589848    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istiod-ca-cert\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589892    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-socket\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.589937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589936    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-data\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.590150 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.589993    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7n2\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-kube-api-access-gz7n2\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.596151 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.596122    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"]
Sep 15 21:23:55.599579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.599533    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.600821 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.600796    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:55.602569 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.602552    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-625609-openshift-default-dockercfg-jlrr7\""
Sep 15 21:23:55.624123 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.624097    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"]
Sep 15 21:23:55.690765 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690727    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istiod-ca-cert\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.690765 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690767    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-socket\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2726b1ad-32a6-400a-a695-18b14674eed5-istio-podinfo\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690805    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-data\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690824    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7n2\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-kube-api-access-gz7n2\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690841    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2726b1ad-32a6-400a-a695-18b14674eed5-istiod-ca-cert\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690860    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-socket\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690877    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-envoy\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690913    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-istio-token\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.690948 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690941    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-podinfo\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690955    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-token\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690971    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-data\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.690996    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-credential-socket\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.691017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-credential-socket\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.691034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pff5q\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-kube-api-access-pff5q\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.691050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-certs\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.691078    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-certs\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.691209 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.691092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-envoy\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.691760 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.691736    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-data\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.692286 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.692264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istiod-ca-cert\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.692382 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.692270    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-credential-socket\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.692574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.692552    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-socket\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.693285 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.693260    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-certs\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.693895 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.693875    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-envoy\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.695320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.695303    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-podinfo\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.714698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.714670    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-token\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.715607 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.715587    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7n2\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-kube-api-access-gz7n2\") pod \"e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") " pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.791794 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-istio-token\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.791965 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-data\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.791965 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791843    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-credential-socket\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.791965 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pff5q\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-kube-api-access-pff5q\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.791965 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791879    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-certs\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.791965 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791950    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2726b1ad-32a6-400a-a695-18b14674eed5-istio-podinfo\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792240 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.791986    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2726b1ad-32a6-400a-a695-18b14674eed5-istiod-ca-cert\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792240 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792013    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-socket\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792240 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-envoy\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792327    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-data\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792531 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792509    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-credential-socket\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792594 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792545    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-certs\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792659    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-socket\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.792913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.792895    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2726b1ad-32a6-400a-a695-18b14674eed5-istiod-ca-cert\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.794625 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.794608    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-envoy\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.794829 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.794812    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2726b1ad-32a6-400a-a695-18b14674eed5-istio-podinfo\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.800525 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.800502    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-istio-token\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.802458 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.802434    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pff5q\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-kube-api-access-pff5q\") pod \"e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") " pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:55.813344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.813316    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:55.882061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.882038    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"]
Sep 15 21:23:55.883909 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:23:55.883880    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6b50c6_518d_4090_87fb_38ea2fd328a0.slice/crio-4aac28d6e898eb009ecd860360d9e146087abf54db89925a85dfc0301a129938 WatchSource:0}: Error finding container 4aac28d6e898eb009ecd860360d9e146087abf54db89925a85dfc0301a129938: Status 404 returned error can't find the container with id 4aac28d6e898eb009ecd860360d9e146087abf54db89925a85dfc0301a129938
Sep 15 21:23:55.886261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.886172    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:55.886355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.886304    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:55.886415 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.886351    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:55.911350 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:55.911328    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:56.093735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.093708    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"]
Sep 15 21:23:56.095566 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:23:56.095537    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f77b46f_ea22_43f8_8105_c25ea9cabf4a.slice/crio-82a29f48a556dce06def72405923ad684f7d5008d8a3b51ecc0c01ad59af1eae WatchSource:0}: Error finding container 82a29f48a556dce06def72405923ad684f7d5008d8a3b51ecc0c01ad59af1eae: Status 404 returned error can't find the container with id 82a29f48a556dce06def72405923ad684f7d5008d8a3b51ecc0c01ad59af1eae
Sep 15 21:23:56.097883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.097849    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:56.097978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.097929    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:56.098039 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.097974    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:56.187929 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.187903    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"]
Sep 15 21:23:56.190032 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:23:56.190003    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2726b1ad_32a6_400a_a695_18b14674eed5.slice/crio-63444e94080f7a114f7f917c1bc11f0349efc48656723a97a8012207757eeffa WatchSource:0}: Error finding container 63444e94080f7a114f7f917c1bc11f0349efc48656723a97a8012207757eeffa: Status 404 returned error can't find the container with id 63444e94080f7a114f7f917c1bc11f0349efc48656723a97a8012207757eeffa
Sep 15 21:23:56.193019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.192980    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:56.193124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.193071    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:56.193124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.193117    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:23:56.493324 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.493287    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" event={"ID":"5f77b46f-ea22-43f8-8105-c25ea9cabf4a","Type":"ContainerStarted","Data":"2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"}
Sep 15 21:23:56.493769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.493349    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" event={"ID":"5f77b46f-ea22-43f8-8105-c25ea9cabf4a","Type":"ContainerStarted","Data":"82a29f48a556dce06def72405923ad684f7d5008d8a3b51ecc0c01ad59af1eae"}
Sep 15 21:23:56.495575 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.495543    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" event={"ID":"2726b1ad-32a6-400a-a695-18b14674eed5","Type":"ContainerStarted","Data":"7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"}
Sep 15 21:23:56.495727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.495601    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" event={"ID":"2726b1ad-32a6-400a-a695-18b14674eed5","Type":"ContainerStarted","Data":"63444e94080f7a114f7f917c1bc11f0349efc48656723a97a8012207757eeffa"}
Sep 15 21:23:56.497403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.497379    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" event={"ID":"2b6b50c6-518d-4090-87fb-38ea2fd328a0","Type":"ContainerStarted","Data":"b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"}
Sep 15 21:23:56.497509 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.497421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" event={"ID":"2b6b50c6-518d-4090-87fb-38ea2fd328a0","Type":"ContainerStarted","Data":"4aac28d6e898eb009ecd860360d9e146087abf54db89925a85dfc0301a129938"}
Sep 15 21:23:56.517400 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.517351    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podStartSLOduration=1.517330659 podStartE2EDuration="1.517330659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:56.515881161 +0000 UTC m=+935.317750105" watchObservedRunningTime="2026-09-15 21:23:56.517330659 +0000 UTC m=+935.319199602"
Sep 15 21:23:56.535343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.535287    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podStartSLOduration=1.535269434 podStartE2EDuration="1.535269434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:56.534541639 +0000 UTC m=+935.336410611" watchObservedRunningTime="2026-09-15 21:23:56.535269434 +0000 UTC m=+935.337138379"
Sep 15 21:23:56.559553 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.559496    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podStartSLOduration=1.559474415 podStartE2EDuration="1.559474415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:23:56.556358197 +0000 UTC m=+935.358227140" watchObservedRunningTime="2026-09-15 21:23:56.559474415 +0000 UTC m=+935.361343362"
Sep 15 21:23:56.600995 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.600969    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:56.606085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.606061    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:56.814060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.813972    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:56.911705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:56.911673    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:57.503386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:57.503352    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:57.504064 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:57.504048    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:23:57.818662 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:57.818590    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:57.916801 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:57.916773    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:58.507497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:58.507463    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:58.507953 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:58.507517    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:23:58.508449 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:58.508426    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:23:58.508549 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:23:58.508486    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:24:02.780618 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.780588    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"]
Sep 15 21:24:02.784147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.784127    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.786469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.786442    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-70880c-serving-cert\""
Sep 15 21:24:02.798778 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.798751    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"]
Sep 15 21:24:02.856598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.856564    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.856759 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.856601    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931a649-057e-4908-aefa-f799e3a56de9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.856759 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.856706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-metrics-certs\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.856759 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.856732    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9js\" (UniqueName: \"kubernetes.io/projected/c931a649-057e-4908-aefa-f799e3a56de9-kube-api-access-nr9js\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.958111 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.958068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.958346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.958131    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931a649-057e-4908-aefa-f799e3a56de9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.958346 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:24:02.958258    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-serving-cert: secret "maas-api-e2e-inf-a-70880c-serving-cert" not found
Sep 15 21:24:02.958346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.958276    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-metrics-certs\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.958346 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:24:02.958326    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls podName:c931a649-057e-4908-aefa-f799e3a56de9 nodeName:}" failed. No retries permitted until 2026-09-15 21:24:03.458307468 +0000 UTC m=+942.260176394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls") pod "maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" (UID: "c931a649-057e-4908-aefa-f799e3a56de9") : secret "maas-api-e2e-inf-a-70880c-serving-cert" not found
Sep 15 21:24:02.958578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.958351    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9js\" (UniqueName: \"kubernetes.io/projected/c931a649-057e-4908-aefa-f799e3a56de9-kube-api-access-nr9js\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.958889 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.958863    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931a649-057e-4908-aefa-f799e3a56de9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.960835 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.960816    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-metrics-certs\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:02.971424 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:02.971397    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9js\" (UniqueName: \"kubernetes.io/projected/c931a649-057e-4908-aefa-f799e3a56de9-kube-api-access-nr9js\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:03.462773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.462744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:03.465375 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.465346    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls\") pod \"maas-api-e2e-inf-a-70880c-6c9c974775-jz57h\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:03.697241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.697185    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:03.894897 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.894866    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"]
Sep 15 21:24:03.898558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.898531    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.900868 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.900849    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-faa4c5-openshift-default-dockercfg-fzwpr\""
Sep 15 21:24:03.910725 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.910700    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"]
Sep 15 21:24:03.966946 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.966908    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-credential-socket\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967106 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.966954    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-envoy\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967106 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istiod-ca-cert\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967106 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967078    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-socket\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967106 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967097    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-token\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-certs\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967274    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-podinfo\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967421 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967364    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-data\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.967421 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.967383    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfs5\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-kube-api-access-2hfs5\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:03.974335 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:03.974262    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"]
Sep 15 21:24:03.978547 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:24:03.978523    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc931a649_057e_4908_aefa_f799e3a56de9.slice/crio-4b1e7e4f3f098d59708415f598ba634cc8e336731e2b4ecd2ddd64be69a4be2a WatchSource:0}: Error finding container 4b1e7e4f3f098d59708415f598ba634cc8e336731e2b4ecd2ddd64be69a4be2a: Status 404 returned error can't find the container with id 4b1e7e4f3f098d59708415f598ba634cc8e336731e2b4ecd2ddd64be69a4be2a
Sep 15 21:24:04.068653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-certs\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.068653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-podinfo\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.068856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068689    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-data\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.068856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068715    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfs5\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-kube-api-access-2hfs5\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.068856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068754    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-credential-socket\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.068856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068788    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-envoy\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068874    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istiod-ca-cert\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068903    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-socket\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.068932    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-token\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.069072    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-certs\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.069137    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-data\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069556 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.069530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-socket\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.069535    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-credential-socket\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.069819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.069700    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istiod-ca-cert\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.071445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.071420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-podinfo\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.072411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.072368    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-envoy\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.081412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.081266    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-token\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.081412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.081380    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfs5\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-kube-api-access-2hfs5\") pod \"e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") " pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.210327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.210274    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:04.486684 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.486656    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"]
Sep 15 21:24:04.488625 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:24:04.488596    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156a2ca3_7c4c_481e_a878_05fbfdbf65f4.slice/crio-4a85b83cd11a68a9b0d25f68ad51fd6180f23ac7ddb50960d378227283c001b0 WatchSource:0}: Error finding container 4a85b83cd11a68a9b0d25f68ad51fd6180f23ac7ddb50960d378227283c001b0: Status 404 returned error can't find the container with id 4a85b83cd11a68a9b0d25f68ad51fd6180f23ac7ddb50960d378227283c001b0
Sep 15 21:24:04.490892 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.490857    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:04.490986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.490923    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:04.490986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.490955    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:04.538283 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.538235    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" event={"ID":"c931a649-057e-4908-aefa-f799e3a56de9","Type":"ContainerStarted","Data":"19e722471a819a89f7a36818c176ed585bc025708659aaf9516773b4633233a6"}
Sep 15 21:24:04.538449 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.538316    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" event={"ID":"c931a649-057e-4908-aefa-f799e3a56de9","Type":"ContainerStarted","Data":"4b1e7e4f3f098d59708415f598ba634cc8e336731e2b4ecd2ddd64be69a4be2a"}
Sep 15 21:24:04.538514 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.538452    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:04.539544 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.539517    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" event={"ID":"156a2ca3-7c4c-481e-a878-05fbfdbf65f4","Type":"ContainerStarted","Data":"4a85b83cd11a68a9b0d25f68ad51fd6180f23ac7ddb50960d378227283c001b0"}
Sep 15 21:24:04.557852 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:04.557798    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" podStartSLOduration=2.148164631 podStartE2EDuration="2.557780407s" totalImagesPullingTime="409.615776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:04.555824583 +0000 UTC m=+943.357693547" watchObservedRunningTime="2026-09-15 21:24:04.557780407 +0000 UTC m=+943.359649350"
Sep 15 21:24:05.227509 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.227468    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"]
Sep 15 21:24:05.231087 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.231066    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.233954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.233930    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-44fbc30a-openshift-default-dockercfg-skfz6\""
Sep 15 21:24:05.248771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.248745    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"]
Sep 15 21:24:05.281060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281033    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-data\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3476bf7c-1e91-415c-a75f-f640c2fa6050-istiod-ca-cert\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281091    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pn47\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-kube-api-access-4pn47\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281108    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-envoy\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-token\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281274    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-certs\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281295    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-credential-socket\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281427 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281335    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-podinfo\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.281427 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.281357    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-socket\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382022    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pn47\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-kube-api-access-4pn47\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-envoy\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382095    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-token\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382135    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-certs\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-credential-socket\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382350    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-podinfo\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382390    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-socket\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382530    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-data\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382552    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-certs\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382558    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3476bf7c-1e91-415c-a75f-f640c2fa6050-istiod-ca-cert\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382778 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382648    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-credential-socket\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382778 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382715    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-socket\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.382872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.382829    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-data\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.383473 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.383450    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3476bf7c-1e91-415c-a75f-f640c2fa6050-istiod-ca-cert\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.384770 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.384747    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-envoy\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.384934 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.384915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-podinfo\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.397260 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.397235    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-token\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.397452 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.397433    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pn47\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-kube-api-access-4pn47\") pod \"e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") " pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.542157 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.542065    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:05.545949 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.545916    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" event={"ID":"156a2ca3-7c4c-481e-a878-05fbfdbf65f4","Type":"ContainerStarted","Data":"cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"}
Sep 15 21:24:05.576119 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.576056    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podStartSLOduration=2.576033707 podStartE2EDuration="2.576033707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:05.575108336 +0000 UTC m=+944.376977301" watchObservedRunningTime="2026-09-15 21:24:05.576033707 +0000 UTC m=+944.377902650"
Sep 15 21:24:05.826591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.826507    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"]
Sep 15 21:24:05.829777 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:24:05.829748    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3476bf7c_1e91_415c_a75f_f640c2fa6050.slice/crio-22dcd684824589ed68e1b6807b925c352c43c51f68355bed827c8d940cc8da09 WatchSource:0}: Error finding container 22dcd684824589ed68e1b6807b925c352c43c51f68355bed827c8d940cc8da09: Status 404 returned error can't find the container with id 22dcd684824589ed68e1b6807b925c352c43c51f68355bed827c8d940cc8da09
Sep 15 21:24:05.832128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.832094    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:05.832218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.832163    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:05.832218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:05.832213    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:06.210802 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.210769    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:06.215782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.215756    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:06.553557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.553454    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" event={"ID":"3476bf7c-1e91-415c-a75f-f640c2fa6050","Type":"ContainerStarted","Data":"97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"}
Sep 15 21:24:06.553557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.553513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" event={"ID":"3476bf7c-1e91-415c-a75f-f640c2fa6050","Type":"ContainerStarted","Data":"22dcd684824589ed68e1b6807b925c352c43c51f68355bed827c8d940cc8da09"}
Sep 15 21:24:06.554051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.553670    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:06.554709 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.554692    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:06.579400 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:06.579339    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podStartSLOduration=1.579323697 podStartE2EDuration="1.579323697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:06.578272739 +0000 UTC m=+945.380141700" watchObservedRunningTime="2026-09-15 21:24:06.579323697 +0000 UTC m=+945.381192639"
Sep 15 21:24:07.542965 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:07.542929    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:07.548339 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:07.548311    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:07.558448 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:07.558417    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:07.559322 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:07.559302    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:10.554920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:10.554888    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:24:12.549678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.549646    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"]
Sep 15 21:24:12.555149 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.555116    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.558044 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.558010    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-faa4c5-serving-cert\""
Sep 15 21:24:12.596218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.582816    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"]
Sep 15 21:24:12.675219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.673976    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.675219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.674048    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-openshift-service-ca\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.675219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.674083    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnqgk\" (UniqueName: \"kubernetes.io/projected/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-kube-api-access-hnqgk\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.675219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.674167    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-metrics-certs\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.775215 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.775080    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-metrics-certs\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.779215 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.775876    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.779215 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.775953    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-openshift-service-ca\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.779215 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.776000    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnqgk\" (UniqueName: \"kubernetes.io/projected/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-kube-api-access-hnqgk\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.779215 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:24:12.776549    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-serving-cert: secret "maas-api-e2e-autores-faa4c5-serving-cert" not found
Sep 15 21:24:12.779215 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:24:12.776639    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls podName:2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26 nodeName:}" failed. No retries permitted until 2026-09-15 21:24:13.276618582 +0000 UTC m=+952.078487505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls") pod "maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" (UID: "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26") : secret "maas-api-e2e-autores-faa4c5-serving-cert" not found
Sep 15 21:24:12.779215 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.777715    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-openshift-service-ca\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.783217 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.781811    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-metrics-certs\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:12.791215 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:12.791020    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnqgk\" (UniqueName: \"kubernetes.io/projected/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-kube-api-access-hnqgk\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:13.291219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:13.288048    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:13.295223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:13.292091    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls\") pod \"maas-api-e2e-autores-faa4c5-5c5fcf545f-np567\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:13.468455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:13.468421    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:13.952392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:13.952255    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"]
Sep 15 21:24:13.955076 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:24:13.954997    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1cc5fe_ce2a_4e7b_b1a9_e6898bcdbe26.slice/crio-cdd51c3793efda1cc7ed8a256787a61e7628e5ef8974e3ac320d3bf08a8c2c63 WatchSource:0}: Error finding container cdd51c3793efda1cc7ed8a256787a61e7628e5ef8974e3ac320d3bf08a8c2c63: Status 404 returned error can't find the container with id cdd51c3793efda1cc7ed8a256787a61e7628e5ef8974e3ac320d3bf08a8c2c63
Sep 15 21:24:13.956666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:13.956634    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:24:14.621336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:14.621290    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" event={"ID":"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26","Type":"ContainerStarted","Data":"395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7"}
Sep 15 21:24:14.621544 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:14.621387    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" event={"ID":"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26","Type":"ContainerStarted","Data":"cdd51c3793efda1cc7ed8a256787a61e7628e5ef8974e3ac320d3bf08a8c2c63"}
Sep 15 21:24:14.622563 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:14.622536    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:14.643351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:14.643294    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" podStartSLOduration=2.105163737 podStartE2EDuration="2.643278928s" totalImagesPullingTime="538.115191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:14.643026218 +0000 UTC m=+953.444895160" watchObservedRunningTime="2026-09-15 21:24:14.643278928 +0000 UTC m=+953.445147870"
Sep 15 21:24:20.665135 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.665096    2551 generic.go:356] "Generic (PLEG): container finished" podID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerID="2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7" exitCode=0
Sep 15 21:24:20.665660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.665152    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" event={"ID":"5f77b46f-ea22-43f8-8105-c25ea9cabf4a","Type":"ContainerDied","Data":"2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"}
Sep 15 21:24:20.665832 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.665819    2551 scope.go:117] "RemoveContainer" containerID="2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"
Sep 15 21:24:20.666718 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.666699    2551 generic.go:356] "Generic (PLEG): container finished" podID="2726b1ad-32a6-400a-a695-18b14674eed5" containerID="7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b" exitCode=0
Sep 15 21:24:20.666807 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.666794    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" event={"ID":"2726b1ad-32a6-400a-a695-18b14674eed5","Type":"ContainerDied","Data":"7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"}
Sep 15 21:24:20.667176 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.667164    2551 scope.go:117] "RemoveContainer" containerID="7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"
Sep 15 21:24:20.667405 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.667367    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.667488 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.667466    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.667545 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.667521    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.669849 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.669814    2551 generic.go:356] "Generic (PLEG): container finished" podID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerID="b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c" exitCode=0
Sep 15 21:24:20.670007 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.669991    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" event={"ID":"2b6b50c6-518d-4090-87fb-38ea2fd328a0","Type":"ContainerDied","Data":"b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"}
Sep 15 21:24:20.670688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.670672    2551 scope.go:117] "RemoveContainer" containerID="b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"
Sep 15 21:24:20.671846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.671013    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.671846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.671099    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.671846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.671176    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.672454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.672358    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.672454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.672438    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:20.672588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:20.672486    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:21.646408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.646371    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:24:21.695450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.693506    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" event={"ID":"5f77b46f-ea22-43f8-8105-c25ea9cabf4a","Type":"ContainerStarted","Data":"44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922"}
Sep 15 21:24:21.698651 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.698616    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" event={"ID":"2726b1ad-32a6-400a-a695-18b14674eed5","Type":"ContainerStarted","Data":"f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3"}
Sep 15 21:24:21.729753 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.729711    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" event={"ID":"2b6b50c6-518d-4090-87fb-38ea2fd328a0","Type":"ContainerStarted","Data":"21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98"}
Sep 15 21:24:21.815077 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.814610    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:24:21.815077 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.814742    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:21.815077 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.814790    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:21.911805 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.911775    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:24:21.912141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.912119    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:21.912352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:21.912328    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:22.602282 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.602242    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:24:22.602715 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.602629    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:22.602888 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.602866    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:22.814019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.813966    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:22.814571 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.814047    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:22.912589 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.912539    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:22.912798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:22.912618    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:23.601984 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:23.601931    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:23.602216 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:23.602013    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:23.814987 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:23.814543    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:23.814987 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:23.814609    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:23.912749 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:23.912704    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:23.912945 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:23.912773    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:24.236717 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.236614    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"]
Sep 15 21:24:24.246698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.246665    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.249572 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.249348    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-779ace-openshift-default-dockercfg-6gr5n\""
Sep 15 21:24:24.343495 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.343462    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"]
Sep 15 21:24:24.371785 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.371748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-certs\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372045 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372024    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c283b47-2030-4956-b896-d6773ec132c6-istiod-ca-cert\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372221 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-credential-socket\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372336    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-istio-token\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c283b47-2030-4956-b896-d6773ec132c6-istio-podinfo\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372579    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-data\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372691 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372680    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-socket\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372797 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372786    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-envoy\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.372917 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.372904    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbmg\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-kube-api-access-lkbmg\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.475840 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.475791    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbmg\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-kube-api-access-lkbmg\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.476504 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.476480    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-certs\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.477108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.477091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c283b47-2030-4956-b896-d6773ec132c6-istiod-ca-cert\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.478100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.478082    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-credential-socket\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.478637 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.478619    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-istio-token\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.479109 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.479091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c283b47-2030-4956-b896-d6773ec132c6-istio-podinfo\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.479729 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.479710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-data\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.480253 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.480226    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-socket\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.480723 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.480704    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-envoy\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.481233 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.480126    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-data\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.488280 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.477987    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c283b47-2030-4956-b896-d6773ec132c6-istiod-ca-cert\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.488677 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.480626    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-socket\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.488823 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.477037    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-certs\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.488978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.478521    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-credential-socket\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.501956 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.501879    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-envoy\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.504834 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.504807    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbmg\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-kube-api-access-lkbmg\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.505012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.504812    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c283b47-2030-4956-b896-d6773ec132c6-istio-podinfo\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.509659 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.509631    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-istio-token\") pod \"e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") " pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.582337 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.581989    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:24.603654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.603606    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:24.603856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.603699    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:24.814576 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.814029    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:24.814576 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.814096    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:24.913152 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.912658    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:24.913152 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:24.912730    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:25.148829 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.146807    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"]
Sep 15 21:24:25.167057 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:24:25.167002    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c283b47_2030_4956_b896_d6773ec132c6.slice/crio-645e7a689e1eaf404913bebb2f23b8d2811bcec182ede0da65030291fa0f090d WatchSource:0}: Error finding container 645e7a689e1eaf404913bebb2f23b8d2811bcec182ede0da65030291fa0f090d: Status 404 returned error can't find the container with id 645e7a689e1eaf404913bebb2f23b8d2811bcec182ede0da65030291fa0f090d
Sep 15 21:24:25.171056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.169771    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:25.171056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.169859    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:25.171056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.169908    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:25.609228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.605265    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:25.609228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.605336    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:25.786220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.783533    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" event={"ID":"5c283b47-2030-4956-b896-d6773ec132c6","Type":"ContainerStarted","Data":"21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"}
Sep 15 21:24:25.786220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.783608    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" event={"ID":"5c283b47-2030-4956-b896-d6773ec132c6","Type":"ContainerStarted","Data":"645e7a689e1eaf404913bebb2f23b8d2811bcec182ede0da65030291fa0f090d"}
Sep 15 21:24:25.813218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.812854    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podStartSLOduration=1.812829331 podStartE2EDuration="1.812829331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:25.809259902 +0000 UTC m=+964.611128874" watchObservedRunningTime="2026-09-15 21:24:25.812829331 +0000 UTC m=+964.614698276"
Sep 15 21:24:25.817219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.814599    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:25.817219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.814664    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:25.912686 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.912641    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:25.912881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:25.912709    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:26.585220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.582550    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:26.596905 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.596868    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:26.607754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.607691    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:26.608328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.608290    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:26.793613 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.793582    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:26.797742 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.797711    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:26.814840 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.814795    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:26.815141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.815092    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:26.913290 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.912147    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:26.913290 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:26.912228    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:27.503232 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.503155    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:24:27.601614 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.601514    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:27.601614 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.601588    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:27.813766 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.813725    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:27.814046 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.814015    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:27.912867 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.912799    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:27.913168 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:27.913121    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:28.507529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.507488    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:24:28.508740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.508717    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:24:28.602108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.601636    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 15 21:24:28.602108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.601717    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 15 21:24:28.816706 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.816615    2551 patch_prober.go:28] interesting pod/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 21:24:28.816944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.816910    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 21:24:28.912644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.912606    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 15 21:24:28.912941 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:28.912914    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 15 21:24:29.605210 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:29.605161    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:24:29.606574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:29.606554    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:24:29.819783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:29.819703    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:24:29.823542 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:29.823512    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:24:29.920798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:29.920767    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:24:29.922536 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:29.922501    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:24:31.838002 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.837922    2551 generic.go:356] "Generic (PLEG): container finished" podID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerID="cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c" exitCode=0
Sep 15 21:24:31.838002 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.837993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" event={"ID":"156a2ca3-7c4c-481e-a878-05fbfdbf65f4","Type":"ContainerDied","Data":"cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"}
Sep 15 21:24:31.838531 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.838434    2551 scope.go:117] "RemoveContainer" containerID="cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"
Sep 15 21:24:31.839791 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.839756    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:31.839896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.839805    2551 generic.go:356] "Generic (PLEG): container finished" podID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerID="97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a" exitCode=0
Sep 15 21:24:31.839896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.839829    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:31.839896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.839850    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" event={"ID":"3476bf7c-1e91-415c-a75f-f640c2fa6050","Type":"ContainerDied","Data":"97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"}
Sep 15 21:24:31.839896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.839863    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:31.840205 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.840176    2551 scope.go:117] "RemoveContainer" containerID="97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"
Sep 15 21:24:31.841246 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.841221    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:31.841308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.841279    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:31.841347 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:31.841316    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:32.847164 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:32.847120    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" event={"ID":"156a2ca3-7c4c-481e-a878-05fbfdbf65f4","Type":"ContainerStarted","Data":"eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb"}
Sep 15 21:24:32.849106 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:32.849079    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" event={"ID":"3476bf7c-1e91-415c-a75f-f640c2fa6050","Type":"ContainerStarted","Data":"348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071"}
Sep 15 21:24:33.212372 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:33.211137    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:33.212599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:33.212446    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 15 21:24:33.212599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:33.212547    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 15 21:24:33.558478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:33.558378    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:33.560798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:33.560324    2551 patch_prober.go:28] interesting pod/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:24:33.560798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:33.560405    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 15 21:24:34.212218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:34.211317    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 15 21:24:34.212218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:34.211382    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 15 21:24:34.542637 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:34.542542    2551 patch_prober.go:28] interesting pod/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:24:34.542637 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:34.542617    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 15 21:24:35.211219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:35.210732    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 15 21:24:35.211219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:35.210801    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 15 21:24:35.542698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:35.542608    2551 patch_prober.go:28] interesting pod/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:24:35.542698 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:35.542668    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 15 21:24:36.215252 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:36.213690    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 15 21:24:36.215252 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:36.213753    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 15 21:24:36.543340 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:36.543255    2551 patch_prober.go:28] interesting pod/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:24:36.543340 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:36.543313    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 15 21:24:36.554062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:36.554032    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:37.211556 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:37.211510    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 15 21:24:37.211767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:37.211599    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 15 21:24:37.543516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:37.543437    2551 patch_prober.go:28] interesting pod/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:24:37.543958 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:37.543934    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 15 21:24:37.558427 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:37.558411    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:38.210818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:38.210778    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 15 21:24:38.211123 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:38.211091    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 15 21:24:38.542782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:38.542701    2551 patch_prober.go:28] interesting pod/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:24:38.542996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:38.542970    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 15 21:24:39.216242 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:39.216183    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:39.217547 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:39.217523    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:24:39.318902 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:39.318869    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"]
Sep 15 21:24:39.319133 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:39.319109    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy" containerID="cri-o://348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071" gracePeriod=30
Sep 15 21:24:44.621157 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.621132    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:44.693609 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693561    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3476bf7c-1e91-415c-a75f-f640c2fa6050-istiod-ca-cert\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.693819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693625    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pn47\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-kube-api-access-4pn47\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.693819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693659    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-data\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.693819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693696    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-podinfo\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.693819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693726    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-certs\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.693819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693798    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-token\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.694069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693874    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-credential-socket\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.694069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693918    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-socket\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.694069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.693983    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-envoy\") pod \"3476bf7c-1e91-415c-a75f-f640c2fa6050\" (UID: \"3476bf7c-1e91-415c-a75f-f640c2fa6050\") "
Sep 15 21:24:44.694845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.694594    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:44.694845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.694750    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:44.694845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.694822    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3476bf7c-1e91-415c-a75f-f640c2fa6050-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:24:44.695408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.695367    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-data" (OuterVolumeSpecName: "istio-data") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:44.695865 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.695842    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:44.697602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.697556    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:24:44.697602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.697570    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-token" (OuterVolumeSpecName: "istio-token") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:44.699032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.699010    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-kube-api-access-4pn47" (OuterVolumeSpecName: "kube-api-access-4pn47") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "kube-api-access-4pn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:24:44.699392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.699374    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3476bf7c-1e91-415c-a75f-f640c2fa6050" (UID: "3476bf7c-1e91-415c-a75f-f640c2fa6050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:24:44.796032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.795998    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796033    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796049    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796063    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796075    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3476bf7c-1e91-415c-a75f-f640c2fa6050-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796086    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pn47\" (UniqueName: \"kubernetes.io/projected/3476bf7c-1e91-415c-a75f-f640c2fa6050-kube-api-access-4pn47\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796096    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796107    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3476bf7c-1e91-415c-a75f-f640c2fa6050-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.796317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.796118    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3476bf7c-1e91-415c-a75f-f640c2fa6050-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:24:44.919890 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.919854    2551 generic.go:356] "Generic (PLEG): container finished" podID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerID="348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071" exitCode=0
Sep 15 21:24:44.920086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.919890    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" event={"ID":"3476bf7c-1e91-415c-a75f-f640c2fa6050","Type":"ContainerDied","Data":"348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071"}
Sep 15 21:24:44.920086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.919908    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"
Sep 15 21:24:44.920086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.919950    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5" event={"ID":"3476bf7c-1e91-415c-a75f-f640c2fa6050","Type":"ContainerDied","Data":"22dcd684824589ed68e1b6807b925c352c43c51f68355bed827c8d940cc8da09"}
Sep 15 21:24:44.920086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.919976    2551 scope.go:117] "RemoveContainer" containerID="348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071"
Sep 15 21:24:44.929883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.929854    2551 scope.go:117] "RemoveContainer" containerID="97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"
Sep 15 21:24:44.938761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.938742    2551 scope.go:117] "RemoveContainer" containerID="348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071"
Sep 15 21:24:44.939014 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:24:44.938993    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071\": container with ID starting with 348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071 not found: ID does not exist" containerID="348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071"
Sep 15 21:24:44.939060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.939024    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071"} err="failed to get container status \"348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071\": rpc error: code = NotFound desc = could not find container \"348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071\": container with ID starting with 348dc10ba3a7602f3e4376c6cc51ba7f5b5e7b60c7554fc9c6420b219f457071 not found: ID does not exist"
Sep 15 21:24:44.939060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.939042    2551 scope.go:117] "RemoveContainer" containerID="97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"
Sep 15 21:24:44.939328 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:24:44.939310    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a\": container with ID starting with 97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a not found: ID does not exist" containerID="97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"
Sep 15 21:24:44.939393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.939337    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a"} err="failed to get container status \"97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a\": rpc error: code = NotFound desc = could not find container \"97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a\": container with ID starting with 97488d41e1b6dab72f9fc62ce7f3961466bb9ef3627aac92893378174868453a not found: ID does not exist"
Sep 15 21:24:44.942772 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.942749    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"]
Sep 15 21:24:44.948069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:44.948049    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-44fbc30a-openshift-default-7c5ffcf786-9tqh5"]
Sep 15 21:24:45.787217 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:45.785969    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" path="/var/lib/kubelet/pods/3476bf7c-1e91-415c-a75f-f640c2fa6050/volumes"
Sep 15 21:24:49.949392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:49.949359    2551 generic.go:356] "Generic (PLEG): container finished" podID="5c283b47-2030-4956-b896-d6773ec132c6" containerID="21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e" exitCode=0
Sep 15 21:24:49.949818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:49.949436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" event={"ID":"5c283b47-2030-4956-b896-d6773ec132c6","Type":"ContainerDied","Data":"21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"}
Sep 15 21:24:49.949912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:49.949896    2551 scope.go:117] "RemoveContainer" containerID="21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"
Sep 15 21:24:49.951287 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:49.951258    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:49.951545 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:49.951326    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:49.951545 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:49.951359    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:24:50.957060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:50.957024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" event={"ID":"5c283b47-2030-4956-b896-d6773ec132c6","Type":"ContainerStarted","Data":"195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e"}
Sep 15 21:24:51.582936 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:51.582849    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:51.583591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:51.583255    2551 patch_prober.go:28] interesting pod/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:24:51.583591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:51.583311    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:24:52.583731 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:52.583691    2551 patch_prober.go:28] interesting pod/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:24:52.584381 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:52.584344    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:24:53.583300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:53.583257    2551 patch_prober.go:28] interesting pod/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:24:53.583627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:53.583590    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:24:54.586219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:54.585627    2551 patch_prober.go:28] interesting pod/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:24:54.586219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:54.585707    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:24:55.583779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:55.583737    2551 patch_prober.go:28] interesting pod/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 15 21:24:55.584149 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:55.584113    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 15 21:24:56.616968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:56.616862    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:56.617736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:56.617313    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:24:56.618778 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:24:56.618759    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:25:37.014952 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.014916    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"]
Sep 15 21:25:37.015548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.015529    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy"
Sep 15 21:25:37.015627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.015552    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy"
Sep 15 21:25:37.015627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.015590    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy"
Sep 15 21:25:37.015627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.015600    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy"
Sep 15 21:25:37.015770 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.015708    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy"
Sep 15 21:25:37.015770 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.015722    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3476bf7c-1e91-415c-a75f-f640c2fa6050" containerName="istio-proxy"
Sep 15 21:25:37.018803 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.018780    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.021982 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.021951    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-625609\"/\"default-dockercfg-dzqv9\""
Sep 15 21:25:37.022125 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.021998    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-625609\"/\"openshift-service-ca.crt\""
Sep 15 21:25:37.022125 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.021951    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-625609\"/\"simulator-w4-625609-kserve-self-signed-certs\""
Sep 15 21:25:37.022125 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.021965    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-625609\"/\"kube-root-ca.crt\""
Sep 15 21:25:37.027394 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.027369    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"]
Sep 15 21:25:37.126383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.126330    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-model-cache\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.126383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.126389    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-tmp-dir\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.126702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.126470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-home\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.126702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.126502    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7a4da-4b35-47e5-b761-831212f95f0b-tls-certs\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.126702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.126523    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-dshm\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.126702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.126594    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxn5\" (UniqueName: \"kubernetes.io/projected/76f7a4da-4b35-47e5-b761-831212f95f0b-kube-api-access-fdxn5\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228085    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-home\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228129    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7a4da-4b35-47e5-b761-831212f95f0b-tls-certs\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228150    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-dshm\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxn5\" (UniqueName: \"kubernetes.io/projected/76f7a4da-4b35-47e5-b761-831212f95f0b-kube-api-access-fdxn5\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228335    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-model-cache\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-tmp-dir\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-home\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228682    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-model-cache\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.228748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.228706    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-tmp-dir\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.230656 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.230633    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-dshm\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.231085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.230915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7a4da-4b35-47e5-b761-831212f95f0b-tls-certs\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.241021 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.240989    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxn5\" (UniqueName: \"kubernetes.io/projected/76f7a4da-4b35-47e5-b761-831212f95f0b-kube-api-access-fdxn5\") pod \"simulator-w4-625609-kserve-977dd4d4f-d4rk6\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") " pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.328681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.328586    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:37.608658 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:37.608387    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"]
Sep 15 21:25:37.610943 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:25:37.610913    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76f7a4da_4b35_47e5_b761_831212f95f0b.slice/crio-aff7e442dd089946578f27526cdc10e47529c60723448916784f805d1afebf6d WatchSource:0}: Error finding container aff7e442dd089946578f27526cdc10e47529c60723448916784f805d1afebf6d: Status 404 returned error can't find the container with id aff7e442dd089946578f27526cdc10e47529c60723448916784f805d1afebf6d
Sep 15 21:25:38.217258 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:38.217221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6" event={"ID":"76f7a4da-4b35-47e5-b761-831212f95f0b","Type":"ContainerStarted","Data":"bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c"}
Sep 15 21:25:38.217643 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:38.217268    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6" event={"ID":"76f7a4da-4b35-47e5-b761-831212f95f0b","Type":"ContainerStarted","Data":"aff7e442dd089946578f27526cdc10e47529c60723448916784f805d1afebf6d"}
Sep 15 21:25:38.233730 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:38.233663    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6" podStartSLOduration=2.233641895 podStartE2EDuration="2.233641895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:38.232047328 +0000 UTC m=+1037.033916273" watchObservedRunningTime="2026-09-15 21:25:38.233641895 +0000 UTC m=+1037.035510838"
Sep 15 21:25:47.329139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:47.329103    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:47.329686 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:47.329158    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:47.341558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:47.341526    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:25:48.284269 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:25:48.284230    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:26:19.231929 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.231851    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"]
Sep 15 21:26:19.234621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.234604    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.237135 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.237113    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-70880c\"/\"openshift-service-ca.crt\""
Sep 15 21:26:19.237693 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.237672    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-70880c\"/\"test-model-70880c-kserve-self-signed-certs\""
Sep 15 21:26:19.237809 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.237743    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-70880c\"/\"kube-root-ca.crt\""
Sep 15 21:26:19.237877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.237847    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-70880c\"/\"default-dockercfg-v8m78\""
Sep 15 21:26:19.244407 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.244382    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"]
Sep 15 21:26:19.259543 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.259513    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-model-cache\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.259722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.259586    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tls-certs\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.259722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.259620    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tmp-dir\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.259722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.259649    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-home\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.259722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.259722    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgj2x\" (UniqueName: \"kubernetes.io/projected/f9a5f552-fc84-45af-9cd6-7957736e9dc5-kube-api-access-kgj2x\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.259917 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.259787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-dshm\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.360701 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.360655    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgj2x\" (UniqueName: \"kubernetes.io/projected/f9a5f552-fc84-45af-9cd6-7957736e9dc5-kube-api-access-kgj2x\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.360872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.360746    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-dshm\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.360872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.360810    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-model-cache\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.360872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.360867    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tls-certs\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.361015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.360897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tmp-dir\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.361015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.360920    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-home\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.361297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.361276    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-model-cache\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.361297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.361291    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-home\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.361436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.361346    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tmp-dir\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.363201 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.363169    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-dshm\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.363449 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.363433    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tls-certs\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.369042 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.369011    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgj2x\" (UniqueName: \"kubernetes.io/projected/f9a5f552-fc84-45af-9cd6-7957736e9dc5-kube-api-access-kgj2x\") pod \"test-model-70880c-kserve-689967bc69-b4989\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") " pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.546923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.546835    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:19.820240 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:19.820212    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"]
Sep 15 21:26:20.444009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:20.443975    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989" event={"ID":"f9a5f552-fc84-45af-9cd6-7957736e9dc5","Type":"ContainerStarted","Data":"602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add"}
Sep 15 21:26:20.444009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:20.444013    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989" event={"ID":"f9a5f552-fc84-45af-9cd6-7957736e9dc5","Type":"ContainerStarted","Data":"8ecb5f5207079c6e4bda82fcfd0a893e47c43c4339e72b10a367db593214b067"}
Sep 15 21:26:20.461936 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:20.461871    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989" podStartSLOduration=1.46185331 podStartE2EDuration="1.46185331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:20.460049939 +0000 UTC m=+1079.261918899" watchObservedRunningTime="2026-09-15 21:26:20.46185331 +0000 UTC m=+1079.263722295"
Sep 15 21:26:29.547621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:29.547570    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:29.548306 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:29.547645    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:29.567445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:29.567412    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:30.528502 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:30.528212    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:26:52.670345 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:52.670314    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"]
Sep 15 21:26:52.670931 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:52.670571    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" podUID="2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" containerName="maas-api" containerID="cri-o://395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7" gracePeriod=30
Sep 15 21:26:52.929167 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:52.929106    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:26:53.117697 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.117663    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnqgk\" (UniqueName: \"kubernetes.io/projected/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-kube-api-access-hnqgk\") pod \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") "
Sep 15 21:26:53.117897 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.117713    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-metrics-certs\") pod \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") "
Sep 15 21:26:53.117897 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.117841    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-openshift-service-ca\") pod \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") "
Sep 15 21:26:53.118018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.117920    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls\") pod \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\" (UID: \"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26\") "
Sep 15 21:26:53.118212 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.118158    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" (UID: "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:26:53.118388 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.118373    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-openshift-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:26:53.120806 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.120778    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-kube-api-access-hnqgk" (OuterVolumeSpecName: "kube-api-access-hnqgk") pod "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" (UID: "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26"). InnerVolumeSpecName "kube-api-access-hnqgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:53.120806 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.120784    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" (UID: "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:53.120978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.120810    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" (UID: "2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:53.219564 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.219473    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnqgk\" (UniqueName: \"kubernetes.io/projected/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-kube-api-access-hnqgk\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:26:53.219564 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.219502    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-metrics-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:26:53.219564 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.219511    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26-maas-api-tls\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:26:53.629352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.629263    2551 generic.go:356] "Generic (PLEG): container finished" podID="2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" containerID="395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7" exitCode=0
Sep 15 21:26:53.629352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.629318    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"
Sep 15 21:26:53.629352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.629337    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" event={"ID":"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26","Type":"ContainerDied","Data":"395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7"}
Sep 15 21:26:53.629580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.629379    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567" event={"ID":"2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26","Type":"ContainerDied","Data":"cdd51c3793efda1cc7ed8a256787a61e7628e5ef8974e3ac320d3bf08a8c2c63"}
Sep 15 21:26:53.629580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.629401    2551 scope.go:117] "RemoveContainer" containerID="395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7"
Sep 15 21:26:53.639049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.639033    2551 scope.go:117] "RemoveContainer" containerID="395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7"
Sep 15 21:26:53.639342 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:26:53.639321    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7\": container with ID starting with 395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7 not found: ID does not exist" containerID="395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7"
Sep 15 21:26:53.639384 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.639352    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7"} err="failed to get container status \"395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7\": rpc error: code = NotFound desc = could not find container \"395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7\": container with ID starting with 395017a0aeaf5bdacaf62ea8b678eb512afae12ce1876586f8a3ee7b55ef31c7 not found: ID does not exist"
Sep 15 21:26:53.650512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.650488    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"]
Sep 15 21:26:53.654250 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.654230    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-faa4c5-5c5fcf545f-np567"]
Sep 15 21:26:53.769281 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:26:53.769246    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" path="/var/lib/kubelet/pods/2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26/volumes"
Sep 15 21:27:35.255567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:35.255525    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"]
Sep 15 21:27:35.256089 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:35.255745    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" containerID="cri-o://eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb" gracePeriod=30
Sep 15 21:27:36.032006 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.031971    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"]
Sep 15 21:27:36.032402 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.032388    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" containerName="maas-api"
Sep 15 21:27:36.032457 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.032403    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" containerName="maas-api"
Sep 15 21:27:36.032493 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.032461    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1cc5fe-ce2a-4e7b-b1a9-e6898bcdbe26" containerName="maas-api"
Sep 15 21:27:36.035593 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.035578    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.039245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.039221    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-928184-openshift-default-dockercfg-fgp96\""
Sep 15 21:27:36.066801 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.066773    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"]
Sep 15 21:27:36.156020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.155983    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-envoy\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156022    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-credential-socket\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156046    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-socket\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156110    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istiod-ca-cert\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156151    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-podinfo\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156272    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-data\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156312    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-certs\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156360    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkxq\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-kube-api-access-ntkxq\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.156425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.156385    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-token\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.257937 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.257899    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-data\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.257945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-certs\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.257999    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkxq\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-kube-api-access-ntkxq\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-token\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-envoy\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258100    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-credential-socket\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258139    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-socket\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258168    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istiod-ca-cert\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258242    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-podinfo\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258458    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-data\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258604    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-certs\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258625    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-socket\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.258853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258675    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-credential-socket\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.259007 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.258920    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istiod-ca-cert\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.260741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.260719    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-envoy\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.260946 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.260929    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-podinfo\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.267042 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.267019    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-token\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.267159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.267104    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkxq\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-kube-api-access-ntkxq\") pod \"e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") " pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.346068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.345965    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:36.625265 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.625239    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"]
Sep 15 21:27:36.627272 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:27:36.627243    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed6f560_3cb7_4fb6_a7b1_24d91c3d85ed.slice/crio-10a229ba9cfea679775ab2db397dafc9ad2c650f373efe5dc8bd2358906cb480 WatchSource:0}: Error finding container 10a229ba9cfea679775ab2db397dafc9ad2c650f373efe5dc8bd2358906cb480: Status 404 returned error can't find the container with id 10a229ba9cfea679775ab2db397dafc9ad2c650f373efe5dc8bd2358906cb480
Sep 15 21:27:36.629998 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.629958    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:27:36.630099 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.630033    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:27:36.630099 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.630069    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:27:36.844593 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.844559    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" event={"ID":"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed","Type":"ContainerStarted","Data":"7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"}
Sep 15 21:27:36.844780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.844621    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" event={"ID":"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed","Type":"ContainerStarted","Data":"10a229ba9cfea679775ab2db397dafc9ad2c650f373efe5dc8bd2358906cb480"}
Sep 15 21:27:36.869178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:36.869122    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podStartSLOduration=1.869103389 podStartE2EDuration="1.869103389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:36.86758297 +0000 UTC m=+1155.669451924" watchObservedRunningTime="2026-09-15 21:27:36.869103389 +0000 UTC m=+1155.670972332"
Sep 15 21:27:37.346166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:37.346128    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:37.351387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:37.351364    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:37.850413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:37.850381    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:37.851480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:37.851458    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:27:39.219446 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:39.219404    2551 patch_prober.go:28] interesting pod/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:27:39.219868 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:39.219477    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:27:40.511015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.510981    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:27:40.603697 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603602    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-credential-socket\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603697 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603658    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-certs\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603705    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hfs5\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-kube-api-access-2hfs5\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603725    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-socket\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603759    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-podinfo\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603786    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-envoy\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603810    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-token\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603847    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istiod-ca-cert\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.603924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603886    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-data\") pod \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\" (UID: \"156a2ca3-7c4c-481e-a878-05fbfdbf65f4\") "
Sep 15 21:27:40.604341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.603959    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:40.604341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604186    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:40.604463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604345    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:40.604463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604388    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:40.604567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604486    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:40.604627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604584    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.604627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604599    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.604627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604614    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.604627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604625    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.605001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.604633    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.606536 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.606495    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:27:40.606660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.606581    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:27:40.606660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.606589    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-kube-api-access-2hfs5" (OuterVolumeSpecName: "kube-api-access-2hfs5") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "kube-api-access-2hfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:27:40.606660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.606592    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "156a2ca3-7c4c-481e-a878-05fbfdbf65f4" (UID: "156a2ca3-7c4c-481e-a878-05fbfdbf65f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:27:40.705324 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.705285    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfs5\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-kube-api-access-2hfs5\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.705324 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.705318    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.705324 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.705328    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.705324 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.705336    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156a2ca3-7c4c-481e-a878-05fbfdbf65f4-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:27:40.868165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.868072    2551 generic.go:356] "Generic (PLEG): container finished" podID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerID="eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb" exitCode=0
Sep 15 21:27:40.868165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.868124    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"
Sep 15 21:27:40.868165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.868129    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" event={"ID":"156a2ca3-7c4c-481e-a878-05fbfdbf65f4","Type":"ContainerDied","Data":"eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb"}
Sep 15 21:27:40.868478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.868217    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54" event={"ID":"156a2ca3-7c4c-481e-a878-05fbfdbf65f4","Type":"ContainerDied","Data":"4a85b83cd11a68a9b0d25f68ad51fd6180f23ac7ddb50960d378227283c001b0"}
Sep 15 21:27:40.868478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.868244    2551 scope.go:117] "RemoveContainer" containerID="eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb"
Sep 15 21:27:40.879832 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.879813    2551 scope.go:117] "RemoveContainer" containerID="cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"
Sep 15 21:27:40.888060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.888042    2551 scope.go:117] "RemoveContainer" containerID="eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb"
Sep 15 21:27:40.888487 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:27:40.888421    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb\": container with ID starting with eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb not found: ID does not exist" containerID="eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb"
Sep 15 21:27:40.888578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.888506    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb"} err="failed to get container status \"eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb\": rpc error: code = NotFound desc = could not find container \"eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb\": container with ID starting with eab4563435f39532b6de852768a5291ee44b03a26c76f189c3a7ab15b1d476eb not found: ID does not exist"
Sep 15 21:27:40.888650 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.888628    2551 scope.go:117] "RemoveContainer" containerID="cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"
Sep 15 21:27:40.889156 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:27:40.889095    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c\": container with ID starting with cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c not found: ID does not exist" containerID="cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"
Sep 15 21:27:40.889156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.889130    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c"} err="failed to get container status \"cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c\": rpc error: code = NotFound desc = could not find container \"cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c\": container with ID starting with cb203e6a69518c17bb852d4c9fafb8fb980634789f24742f0e23dab701dae08c not found: ID does not exist"
Sep 15 21:27:40.891288 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.891268    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"]
Sep 15 21:27:40.897641 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:40.897620    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-faa4c5-openshift-default-7f64cd6f4-d7r54"]
Sep 15 21:27:41.768847 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:41.768805    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" path="/var/lib/kubelet/pods/156a2ca3-7c4c-481e-a878-05fbfdbf65f4/volumes"
Sep 15 21:27:48.909139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.909052    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"]
Sep 15 21:27:48.909708 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.909648    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy"
Sep 15 21:27:48.909708 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.909667    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy"
Sep 15 21:27:48.909708 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.909689    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy"
Sep 15 21:27:48.909708 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.909697    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy"
Sep 15 21:27:48.909914 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.909793    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy"
Sep 15 21:27:48.912070 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.912051    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:48.914482 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.914462    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-779ace\"/\"openshift-service-ca.crt\""
Sep 15 21:27:48.915450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.915421    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-779ace\"/\"kube-root-ca.crt\""
Sep 15 21:27:48.915548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.915483    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-779ace\"/\"default-dockercfg-kpmb5\""
Sep 15 21:27:48.915548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.915510    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-779ace\"/\"test-model-779ace-kserve-self-signed-certs\""
Sep 15 21:27:48.922625 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.922603    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"]
Sep 15 21:27:48.992412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.992379    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-dshm\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:48.992689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.992457    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-model-cache\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:48.992689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.992507    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad28e70b-4637-4a69-843e-239870969716-tls-certs\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:48.992689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.992627    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-home\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:48.992689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.992655    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcmwg\" (UniqueName: \"kubernetes.io/projected/ad28e70b-4637-4a69-843e-239870969716-kube-api-access-gcmwg\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:48.992689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:48.992687    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-tmp-dir\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.093940 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.093902    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-home\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.093940 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.093942    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcmwg\" (UniqueName: \"kubernetes.io/projected/ad28e70b-4637-4a69-843e-239870969716-kube-api-access-gcmwg\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.093965    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-tmp-dir\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.094017    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-dshm\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.094070    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-model-cache\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.094102    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad28e70b-4637-4a69-843e-239870969716-tls-certs\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.094380    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-home\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094502 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.094460    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-tmp-dir\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.094502 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.094490    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-model-cache\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.096502 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.096469    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-dshm\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.096748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.096731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad28e70b-4637-4a69-843e-239870969716-tls-certs\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.104126 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.104103    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcmwg\" (UniqueName: \"kubernetes.io/projected/ad28e70b-4637-4a69-843e-239870969716-kube-api-access-gcmwg\") pod \"test-model-779ace-kserve-74d4c599df-8srfw\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") " pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.222370 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.222282    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:49.493918 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.493892    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"]
Sep 15 21:27:49.495488 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:27:49.495461    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad28e70b_4637_4a69_843e_239870969716.slice/crio-2f79694d831e24dbca9425c83a5809c8db3d116e7e89985988be5749691f23d9 WatchSource:0}: Error finding container 2f79694d831e24dbca9425c83a5809c8db3d116e7e89985988be5749691f23d9: Status 404 returned error can't find the container with id 2f79694d831e24dbca9425c83a5809c8db3d116e7e89985988be5749691f23d9
Sep 15 21:27:49.915272 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.915237    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw" event={"ID":"ad28e70b-4637-4a69-843e-239870969716","Type":"ContainerStarted","Data":"57289cea4f6249a4d2cbfd59c58537dc90cdb3b7590db41052f8c52e4436cbd7"}
Sep 15 21:27:49.915671 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.915288    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw" event={"ID":"ad28e70b-4637-4a69-843e-239870969716","Type":"ContainerStarted","Data":"2f79694d831e24dbca9425c83a5809c8db3d116e7e89985988be5749691f23d9"}
Sep 15 21:27:49.933989 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:49.933943    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw" podStartSLOduration=1.933930112 podStartE2EDuration="1.933930112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:49.932905673 +0000 UTC m=+1168.734774627" watchObservedRunningTime="2026-09-15 21:27:49.933930112 +0000 UTC m=+1168.735799055"
Sep 15 21:27:59.222823 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:59.222789    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:59.222823 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:59.222835    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:59.235438 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:59.235412    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:27:59.975850 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:27:59.975823    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:28:02.983567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:02.983529    2551 generic.go:356] "Generic (PLEG): container finished" podID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerID="7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67" exitCode=0
Sep 15 21:28:02.984029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:02.983592    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" event={"ID":"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed","Type":"ContainerDied","Data":"7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"}
Sep 15 21:28:02.984079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:02.984062    2551 scope.go:117] "RemoveContainer" containerID="7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"
Sep 15 21:28:02.985373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:02.985333    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:28:02.985440 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:02.985418    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:28:02.985476 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:02.985458    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:28:03.995604 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:03.995557    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" event={"ID":"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed","Type":"ContainerStarted","Data":"b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf"}
Sep 15 21:28:04.347067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:04.346964    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:28:04.348471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:04.348442    2551 patch_prober.go:28] interesting pod/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:28:04.348626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:04.348496    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 21:28:05.346923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:05.346882    2551 patch_prober.go:28] interesting pod/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:28:05.347346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:05.346947    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 21:28:06.346575 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:06.346534    2551 patch_prober.go:28] interesting pod/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:28:06.346766 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:06.346597    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 21:28:07.346534 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:07.346499    2551 patch_prober.go:28] interesting pod/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:28:07.346945 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:07.346554    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 21:28:07.850513 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:07.850482    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:28:08.346532 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:08.346495    2551 patch_prober.go:28] interesting pod/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 21:28:08.346919 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:08.346550    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 21:28:09.351148 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:09.351113    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:28:09.352291 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:09.352266    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:28:56.827012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.826970    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"]
Sep 15 21:28:56.827670 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.827651    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="156a2ca3-7c4c-481e-a878-05fbfdbf65f4" containerName="istio-proxy"
Sep 15 21:28:56.829825 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.829805    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:56.833208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.833174    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-928184\"/\"openshift-service-ca.crt\""
Sep 15 21:28:56.833541 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.833508    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-928184\"/\"default-dockercfg-x4nbz\""
Sep 15 21:28:56.833631 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.833544    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-928184\"/\"kube-root-ca.crt\""
Sep 15 21:28:56.833631 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.833548    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-928184\"/\"explicit-model-928184-kserve-self-signed-certs\""
Sep 15 21:28:56.840033 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.840011    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"]
Sep 15 21:28:56.918263 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.918223    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-model-cache\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:56.918263 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.918267    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5whg\" (UniqueName: \"kubernetes.io/projected/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-kube-api-access-g5whg\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:56.918482 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.918297    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tls-certs\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:56.918482 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.918396    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-dshm\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:56.918550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.918488    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-home\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:56.918588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:56.918575    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tmp-dir\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019447 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019410    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tmp-dir\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019456    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-model-cache\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5whg\" (UniqueName: \"kubernetes.io/projected/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-kube-api-access-g5whg\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019515    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tls-certs\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-dshm\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-home\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019906 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019875    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tmp-dir\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.019966 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.019909    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-model-cache\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.020046 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.020026    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-home\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.022003 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.021978    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-dshm\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.022240 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.022212    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tls-certs\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.027663 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.027634    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5whg\" (UniqueName: \"kubernetes.io/projected/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-kube-api-access-g5whg\") pod \"explicit-model-928184-kserve-7765d9fbdf-29pzq\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") " pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.139911 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.139828    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:28:57.415298 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:57.415272    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"]
Sep 15 21:28:57.417553 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:28:57.417523    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b6a3ba4_7e34_419b_ad0e_b414e11cb032.slice/crio-ef9dee0bb3305fbd5e9c6473a82d0e24650fd9f2661ec0ac3920cb418ced08a9 WatchSource:0}: Error finding container ef9dee0bb3305fbd5e9c6473a82d0e24650fd9f2661ec0ac3920cb418ced08a9: Status 404 returned error can't find the container with id ef9dee0bb3305fbd5e9c6473a82d0e24650fd9f2661ec0ac3920cb418ced08a9
Sep 15 21:28:58.279068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:58.279036    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq" event={"ID":"4b6a3ba4-7e34-419b-ad0e-b414e11cb032","Type":"ContainerStarted","Data":"f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89"}
Sep 15 21:28:58.279068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:58.279075    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq" event={"ID":"4b6a3ba4-7e34-419b-ad0e-b414e11cb032","Type":"ContainerStarted","Data":"ef9dee0bb3305fbd5e9c6473a82d0e24650fd9f2661ec0ac3920cb418ced08a9"}
Sep 15 21:28:58.306848 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:28:58.306788    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq" podStartSLOduration=2.306769801 podStartE2EDuration="2.306769801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:58.303914664 +0000 UTC m=+1237.105783606" watchObservedRunningTime="2026-09-15 21:28:58.306769801 +0000 UTC m=+1237.108638753"
Sep 15 21:29:07.140815 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:07.140777    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:29:07.141353 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:07.141113    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:29:07.153791 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:07.153767    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:29:07.344855 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:07.344827    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:29:30.618880 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:30.618791    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"]
Sep 15 21:29:35.583649 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.583613    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"]
Sep 15 21:29:35.586574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.586555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.588911 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.588893    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a3aa55-openshift-default-dockercfg-l7fwh\""
Sep 15 21:29:35.601607 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.601582    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"]
Sep 15 21:29:35.616951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.616918    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-certs\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.616954    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-credential-socket\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.616981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-token\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.617015    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.617044    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-envoy\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617379 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.617148    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-socket\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617379 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.617184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdtm\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-kube-api-access-swdtm\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617379 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.617273    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-podinfo\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.617379 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.617305    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-data\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.718677 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.718633    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-socket\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.718986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.718957    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdtm\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-kube-api-access-swdtm\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719035    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-podinfo\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719036    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-socket\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719083    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-data\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719146    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-certs\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719171    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-credential-socket\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719224    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-token\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719246    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719287    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-envoy\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.719689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719663    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-data\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.720009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.719982    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-credential-socket\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.720136 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.720024    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-certs\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.720280 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.720252    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.722474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.722453    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-envoy\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.722804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.722788    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-podinfo\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.726864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.726846    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdtm\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-kube-api-access-swdtm\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.728121 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.728099    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-token\") pod \"e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") " pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:35.896840 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:35.896744    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:36.176079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.176050    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"]
Sep 15 21:29:36.178061 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:29:36.178032    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9680b161_7c0c_4fb1_8784_66a6a0ccd9d1.slice/crio-48096d8ab4d9309c8fc4ccbefd5a726ffbb7ded77b9cf360f9cb23e64207308c WatchSource:0}: Error finding container 48096d8ab4d9309c8fc4ccbefd5a726ffbb7ded77b9cf360f9cb23e64207308c: Status 404 returned error can't find the container with id 48096d8ab4d9309c8fc4ccbefd5a726ffbb7ded77b9cf360f9cb23e64207308c
Sep 15 21:29:36.180999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.180965    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:29:36.181066 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.181037    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:29:36.181103 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.181074    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:29:36.476823 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.476734    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" event={"ID":"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1","Type":"ContainerStarted","Data":"ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"}
Sep 15 21:29:36.476823 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.476798    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" event={"ID":"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1","Type":"ContainerStarted","Data":"48096d8ab4d9309c8fc4ccbefd5a726ffbb7ded77b9cf360f9cb23e64207308c"}
Sep 15 21:29:36.498361 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.498310    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podStartSLOduration=1.49829326 podStartE2EDuration="1.49829326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:36.496017932 +0000 UTC m=+1275.297886922" watchObservedRunningTime="2026-09-15 21:29:36.49829326 +0000 UTC m=+1275.300162202"
Sep 15 21:29:36.897722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:36.897632    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:37.902584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:37.902556    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:38.486184 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:38.486155    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:38.487115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:38.487098    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:29:45.638825 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:45.638776    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq" podUID="4b6a3ba4-7e34-419b-ad0e-b414e11cb032" containerName="main" containerID="cri-o://f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89" gracePeriod=45
Sep 15 21:29:45.896953 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:45.896893    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:29:46.031385 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031350    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-model-cache\") pod \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") "
Sep 15 21:29:46.031580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031397    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-dshm\") pod \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") "
Sep 15 21:29:46.031580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031465    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-home\") pod \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") "
Sep 15 21:29:46.031580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031547    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5whg\" (UniqueName: \"kubernetes.io/projected/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-kube-api-access-g5whg\") pod \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") "
Sep 15 21:29:46.031743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031618    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tls-certs\") pod \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") "
Sep 15 21:29:46.031743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031645    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tmp-dir\") pod \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\" (UID: \"4b6a3ba4-7e34-419b-ad0e-b414e11cb032\") "
Sep 15 21:29:46.031743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031611    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b6a3ba4-7e34-419b-ad0e-b414e11cb032" (UID: "4b6a3ba4-7e34-419b-ad0e-b414e11cb032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:46.031743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031669    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-home" (OuterVolumeSpecName: "home") pod "4b6a3ba4-7e34-419b-ad0e-b414e11cb032" (UID: "4b6a3ba4-7e34-419b-ad0e-b414e11cb032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:46.031999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031956    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b6a3ba4-7e34-419b-ad0e-b414e11cb032" (UID: "4b6a3ba4-7e34-419b-ad0e-b414e11cb032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:46.031999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031973    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:46.031999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.031990    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:46.033939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.033914    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-dshm" (OuterVolumeSpecName: "dshm") pod "4b6a3ba4-7e34-419b-ad0e-b414e11cb032" (UID: "4b6a3ba4-7e34-419b-ad0e-b414e11cb032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:46.034047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.033917    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b6a3ba4-7e34-419b-ad0e-b414e11cb032" (UID: "4b6a3ba4-7e34-419b-ad0e-b414e11cb032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:46.034047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.033952    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-kube-api-access-g5whg" (OuterVolumeSpecName: "kube-api-access-g5whg") pod "4b6a3ba4-7e34-419b-ad0e-b414e11cb032" (UID: "4b6a3ba4-7e34-419b-ad0e-b414e11cb032"). InnerVolumeSpecName "kube-api-access-g5whg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:46.132842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.132810    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5whg\" (UniqueName: \"kubernetes.io/projected/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-kube-api-access-g5whg\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:46.132842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.132841    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:46.133065 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.132856    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:46.133065 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.132868    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b6a3ba4-7e34-419b-ad0e-b414e11cb032-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:46.532411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.532380    2551 generic.go:356] "Generic (PLEG): container finished" podID="4b6a3ba4-7e34-419b-ad0e-b414e11cb032" containerID="f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89" exitCode=0
Sep 15 21:29:46.532608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.532418    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq" event={"ID":"4b6a3ba4-7e34-419b-ad0e-b414e11cb032","Type":"ContainerDied","Data":"f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89"}
Sep 15 21:29:46.532608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.532477    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq" event={"ID":"4b6a3ba4-7e34-419b-ad0e-b414e11cb032","Type":"ContainerDied","Data":"ef9dee0bb3305fbd5e9c6473a82d0e24650fd9f2661ec0ac3920cb418ced08a9"}
Sep 15 21:29:46.532608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.532497    2551 scope.go:117] "RemoveContainer" containerID="f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89"
Sep 15 21:29:46.532608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.532435    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"
Sep 15 21:29:46.542173 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.542156    2551 scope.go:117] "RemoveContainer" containerID="f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89"
Sep 15 21:29:46.542512 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:29:46.542495    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89\": container with ID starting with f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89 not found: ID does not exist" containerID="f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89"
Sep 15 21:29:46.542572 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.542520    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89"} err="failed to get container status \"f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89\": rpc error: code = NotFound desc = could not find container \"f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89\": container with ID starting with f394bb696c582b2b15b00af00c52667d1a82a9e3034ca8a806c7bfeeae978b89 not found: ID does not exist"
Sep 15 21:29:46.555218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.555179    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"]
Sep 15 21:29:46.557576 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:46.557552    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-928184/explicit-model-928184-kserve-7765d9fbdf-29pzq"]
Sep 15 21:29:47.769165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:47.769127    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6a3ba4-7e34-419b-ad0e-b414e11cb032" path="/var/lib/kubelet/pods/4b6a3ba4-7e34-419b-ad0e-b414e11cb032/volumes"
Sep 15 21:29:51.122173 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:51.122140    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"]
Sep 15 21:29:51.122578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:51.122490    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" containerID="cri-o://b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf" gracePeriod=30
Sep 15 21:29:54.353972 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:54.353938    2551 patch_prober.go:28] interesting pod/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:29:54.354376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:54.354010    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:29:56.406085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.406058    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:29:56.542399 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542353    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-socket\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542399 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542400    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-token\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542432    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-envoy\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542472    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-credential-socket\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542496    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istiod-ca-cert\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542521    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-podinfo\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542546    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-certs\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542636    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:56.542957 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542695    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntkxq\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-kube-api-access-ntkxq\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542957 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542743    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-data\") pod \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\" (UID: \"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed\") "
Sep 15 21:29:56.542957 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542744    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:56.542957 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542830    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:56.542957 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.542855    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:56.543247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.543093    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:56.543311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.543294    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.543368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.543313    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.543368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.543327    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.543368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.543345    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.543368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.543358    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.544889 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.544858    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:29:56.544999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.544928    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-kube-api-access-ntkxq" (OuterVolumeSpecName: "kube-api-access-ntkxq") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "kube-api-access-ntkxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:56.545269 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.545248    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:29:56.545331 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.545294    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" (UID: "2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:56.597449 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.597416    2551 generic.go:356] "Generic (PLEG): container finished" podID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerID="b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf" exitCode=0
Sep 15 21:29:56.597634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.597464    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"
Sep 15 21:29:56.597634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.597510    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" event={"ID":"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed","Type":"ContainerDied","Data":"b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf"}
Sep 15 21:29:56.597634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.597565    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d" event={"ID":"2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed","Type":"ContainerDied","Data":"10a229ba9cfea679775ab2db397dafc9ad2c650f373efe5dc8bd2358906cb480"}
Sep 15 21:29:56.597634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.597591    2551 scope.go:117] "RemoveContainer" containerID="b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf"
Sep 15 21:29:56.607385 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.607367    2551 scope.go:117] "RemoveContainer" containerID="7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"
Sep 15 21:29:56.615626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.615608    2551 scope.go:117] "RemoveContainer" containerID="b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf"
Sep 15 21:29:56.615939 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:29:56.615916    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf\": container with ID starting with b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf not found: ID does not exist" containerID="b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf"
Sep 15 21:29:56.615995 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.615946    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf"} err="failed to get container status \"b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf\": rpc error: code = NotFound desc = could not find container \"b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf\": container with ID starting with b8273724fe744dfdedfafe65ff8d5ad109555dd7267c6770391f26f91acf37cf not found: ID does not exist"
Sep 15 21:29:56.615995 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.615963    2551 scope.go:117] "RemoveContainer" containerID="7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"
Sep 15 21:29:56.616339 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:29:56.616320    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67\": container with ID starting with 7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67 not found: ID does not exist" containerID="7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"
Sep 15 21:29:56.616401 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.616344    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67"} err="failed to get container status \"7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67\": rpc error: code = NotFound desc = could not find container \"7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67\": container with ID starting with 7e2448f06c66714b7436a2d90ebd829d9000b65d852df8d0f33c7cfc23010b67 not found: ID does not exist"
Sep 15 21:29:56.621880 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.621858    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"]
Sep 15 21:29:56.625813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.625789    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-928184-openshift-default-5b478f8d6d-rg86d"]
Sep 15 21:29:56.644557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.644490    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.644557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.644512    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.644557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.644521    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:56.644557 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:56.644530    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkxq\" (UniqueName: \"kubernetes.io/projected/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed-kube-api-access-ntkxq\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:29:57.770128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:29:57.770082    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" path="/var/lib/kubelet/pods/2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed/volumes"
Sep 15 21:30:00.211348 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211314    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"]
Sep 15 21:30:00.211752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211686    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy"
Sep 15 21:30:00.211752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211698    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy"
Sep 15 21:30:00.211752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211706    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6a3ba4-7e34-419b-ad0e-b414e11cb032" containerName="main"
Sep 15 21:30:00.211752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211712    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6a3ba4-7e34-419b-ad0e-b414e11cb032" containerName="main"
Sep 15 21:30:00.211752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211726    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy"
Sep 15 21:30:00.211752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211731    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy"
Sep 15 21:30:00.211940 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211795    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy"
Sep 15 21:30:00.211940 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211804    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed6f560-3cb7-4fb6-a7b1-24d91c3d85ed" containerName="istio-proxy"
Sep 15 21:30:00.211940 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.211812    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6a3ba4-7e34-419b-ad0e-b414e11cb032" containerName="main"
Sep 15 21:30:00.215453 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.215437    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"
Sep 15 21:30:00.230852 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.230833    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:30:00.235774 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.235752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"
Sep 15 21:30:00.252398 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.252370    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"]
Sep 15 21:30:00.261521 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.261475    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"]
Sep 15 21:30:00.264623 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.264603    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"
Sep 15 21:30:00.321294 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.320481    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"
Sep 15 21:30:00.322079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.322055    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"]
Sep 15 21:30:00.552873 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.552792    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"]
Sep 15 21:30:00.555974 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:00.555948    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5c797a_f974_4c79_9d46_3f662969df21.slice/crio-ce63871b6c7aef6d37f37ae602b902a1c75626b4af90ee63c8cfd16ee3d511d0 WatchSource:0}: Error finding container ce63871b6c7aef6d37f37ae602b902a1c75626b4af90ee63c8cfd16ee3d511d0: Status 404 returned error can't find the container with id ce63871b6c7aef6d37f37ae602b902a1c75626b4af90ee63c8cfd16ee3d511d0
Sep 15 21:30:00.558599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.558579    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:30:00.620487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.620453    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr" event={"ID":"be5c797a-f974-4c79-9d46-3f662969df21","Type":"ContainerStarted","Data":"ce63871b6c7aef6d37f37ae602b902a1c75626b4af90ee63c8cfd16ee3d511d0"}
Sep 15 21:30:00.630373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:00.630299    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"]
Sep 15 21:30:00.632587 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:00.632561    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7e424e_cc9f_48b8_95cc_8715e46d07fb.slice/crio-f5907097ef9dba02264ee7e3f9705c25d4cd9fa5e970d0051e3946e7b5a521c8 WatchSource:0}: Error finding container f5907097ef9dba02264ee7e3f9705c25d4cd9fa5e970d0051e3946e7b5a521c8: Status 404 returned error can't find the container with id f5907097ef9dba02264ee7e3f9705c25d4cd9fa5e970d0051e3946e7b5a521c8
Sep 15 21:30:01.629112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:01.629073    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt" event={"ID":"4b7e424e-cc9f-48b8-95cc-8715e46d07fb","Type":"ContainerStarted","Data":"f5907097ef9dba02264ee7e3f9705c25d4cd9fa5e970d0051e3946e7b5a521c8"}
Sep 15 21:30:03.644242 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:03.644015    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr" event={"ID":"be5c797a-f974-4c79-9d46-3f662969df21","Type":"ContainerStarted","Data":"66b8f135d0c2b4d90ecb31ae4eb832d65413aeee0b69e2159cb3a3b686a82a0b"}
Sep 15 21:30:03.661472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:03.661419    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr" podStartSLOduration=1.390509453 podStartE2EDuration="3.661402133s" totalImagesPullingTime="2.27089268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:03.660176469 +0000 UTC m=+1302.462045440" watchObservedRunningTime="2026-09-15 21:30:03.661402133 +0000 UTC m=+1302.463271075"
Sep 15 21:30:04.651743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:04.651706    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt" event={"ID":"4b7e424e-cc9f-48b8-95cc-8715e46d07fb","Type":"ContainerStarted","Data":"9813c51c58a81012402b5ece04e5fca295bd15d0e735e57853734dbbb224d1bc"}
Sep 15 21:30:04.653506 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:04.653482    2551 generic.go:356] "Generic (PLEG): container finished" podID="be5c797a-f974-4c79-9d46-3f662969df21" containerID="66b8f135d0c2b4d90ecb31ae4eb832d65413aeee0b69e2159cb3a3b686a82a0b" exitCode=0
Sep 15 21:30:04.653617 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:04.653549    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr" event={"ID":"be5c797a-f974-4c79-9d46-3f662969df21","Type":"ContainerDied","Data":"66b8f135d0c2b4d90ecb31ae4eb832d65413aeee0b69e2159cb3a3b686a82a0b"}
Sep 15 21:30:04.671625 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:04.671568    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt" podStartSLOduration=1.559003081 podStartE2EDuration="4.671550129s" totalImagesPullingTime="3.112547048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:04.669921644 +0000 UTC m=+1303.471790584" watchObservedRunningTime="2026-09-15 21:30:04.671550129 +0000 UTC m=+1303.473419071"
Sep 15 21:30:05.660425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.660390    2551 generic.go:356] "Generic (PLEG): container finished" podID="4b7e424e-cc9f-48b8-95cc-8715e46d07fb" containerID="9813c51c58a81012402b5ece04e5fca295bd15d0e735e57853734dbbb224d1bc" exitCode=0
Sep 15 21:30:05.660887 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.660476    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt" event={"ID":"4b7e424e-cc9f-48b8-95cc-8715e46d07fb","Type":"ContainerDied","Data":"9813c51c58a81012402b5ece04e5fca295bd15d0e735e57853734dbbb224d1bc"}
Sep 15 21:30:05.662180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.662111    2551 generic.go:356] "Generic (PLEG): container finished" podID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerID="ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf" exitCode=0
Sep 15 21:30:05.662320 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.662185    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" event={"ID":"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1","Type":"ContainerDied","Data":"ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"}
Sep 15 21:30:05.662969 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.662686    2551 scope.go:117] "RemoveContainer" containerID="ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"
Sep 15 21:30:05.664154 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.664111    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:05.664257 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.664237    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:05.664309 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.664294    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:05.828656 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:05.828628    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"
Sep 15 21:30:06.669201 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.669161    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" event={"ID":"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1","Type":"ContainerStarted","Data":"4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57"}
Sep 15 21:30:06.670741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.670720    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr"
Sep 15 21:30:06.670863 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.670767    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825130-282nr" event={"ID":"be5c797a-f974-4c79-9d46-3f662969df21","Type":"ContainerDied","Data":"ce63871b6c7aef6d37f37ae602b902a1c75626b4af90ee63c8cfd16ee3d511d0"}
Sep 15 21:30:06.670863 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.670807    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce63871b6c7aef6d37f37ae602b902a1c75626b4af90ee63c8cfd16ee3d511d0"
Sep 15 21:30:06.803179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.803155    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"
Sep 15 21:30:06.897718 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.897633    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:30:06.899082 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.899043    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:30:06.899245 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:06.899113    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:30:07.682442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:07.682405    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"
Sep 15 21:30:07.693339 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:07.693283    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt" event={"ID":"4b7e424e-cc9f-48b8-95cc-8715e46d07fb","Type":"ContainerDied","Data":"f5907097ef9dba02264ee7e3f9705c25d4cd9fa5e970d0051e3946e7b5a521c8"}
Sep 15 21:30:07.693553 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:07.693372    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5907097ef9dba02264ee7e3f9705c25d4cd9fa5e970d0051e3946e7b5a521c8"
Sep 15 21:30:07.897510 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:07.897468    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:30:07.897705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:07.897528    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:30:08.486484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:08.486437    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:30:08.897487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:08.897393    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:30:08.897487 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:08.897459    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:30:09.119235 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119173    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"]
Sep 15 21:30:09.119877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119756    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5c797a-f974-4c79-9d46-3f662969df21" containerName="cleanup"
Sep 15 21:30:09.119877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119777    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5c797a-f974-4c79-9d46-3f662969df21" containerName="cleanup"
Sep 15 21:30:09.119877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119795    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7e424e-cc9f-48b8-95cc-8715e46d07fb" containerName="cleanup"
Sep 15 21:30:09.119877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119803    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7e424e-cc9f-48b8-95cc-8715e46d07fb" containerName="cleanup"
Sep 15 21:30:09.120118 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119887    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7e424e-cc9f-48b8-95cc-8715e46d07fb" containerName="cleanup"
Sep 15 21:30:09.120118 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.119898    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5c797a-f974-4c79-9d46-3f662969df21" containerName="cleanup"
Sep 15 21:30:09.124184 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.124161    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.128982 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.128957    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9aeead-openshift-default-dockercfg-pqjtk\""
Sep 15 21:30:09.138348 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.138325    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"]
Sep 15 21:30:09.218174 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218141    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-token\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218218    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-data\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218272    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-podinfo\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218296    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75qg\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-kube-api-access-x75qg\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218336    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-certs\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218365    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istiod-ca-cert\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-socket\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218473    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-credential-socket\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.218714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.218509    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-envoy\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319040    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-credential-socket\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319085    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-envoy\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319153    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-token\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319215    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-data\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319273    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-podinfo\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319301    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75qg\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-kube-api-access-x75qg\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319344    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-certs\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319384    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istiod-ca-cert\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319466    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-socket\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319534    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-credential-socket\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319829 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-data\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-socket\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.319959 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.319914    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-certs\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.320247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.320143    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istiod-ca-cert\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.321562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.321542    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-envoy\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.321907 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.321893    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-podinfo\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.333574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.333546    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75qg\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-kube-api-access-x75qg\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.333927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.333906    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-token\") pod \"e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") " pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.434852 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.434811    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:09.739293 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.739262    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"]
Sep 15 21:30:09.741607 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:09.741574    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7bdbac_7830_4314_a910_ddcd3bb9c73f.slice/crio-f5b5e986bc153e3dcd63f933f20f3895a9f7619a9a1873705a74f1fe2eb3ccd4 WatchSource:0}: Error finding container f5b5e986bc153e3dcd63f933f20f3895a9f7619a9a1873705a74f1fe2eb3ccd4: Status 404 returned error can't find the container with id f5b5e986bc153e3dcd63f933f20f3895a9f7619a9a1873705a74f1fe2eb3ccd4
Sep 15 21:30:09.743964 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.743923    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:09.744081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.744013    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:09.744081 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.744058    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:09.897589 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.897495    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:30:09.897589 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:09.897578    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:30:10.708621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:10.708578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" event={"ID":"1b7bdbac-7830-4314-a910-ddcd3bb9c73f","Type":"ContainerStarted","Data":"e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"}
Sep 15 21:30:10.708824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:10.708651    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" event={"ID":"1b7bdbac-7830-4314-a910-ddcd3bb9c73f","Type":"ContainerStarted","Data":"f5b5e986bc153e3dcd63f933f20f3895a9f7619a9a1873705a74f1fe2eb3ccd4"}
Sep 15 21:30:10.733654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:10.733607    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podStartSLOduration=1.7335932010000001 podStartE2EDuration="1.733593201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:10.730874581 +0000 UTC m=+1309.532743523" watchObservedRunningTime="2026-09-15 21:30:10.733593201 +0000 UTC m=+1309.535462144"
Sep 15 21:30:10.897273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:10.897238    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:30:10.897531 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:10.897495    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:30:11.435229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:11.435166    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:11.440260 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:11.440236    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:11.714178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:11.714101    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:11.715180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:11.715164    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:11.897420 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:11.897386    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 15 21:30:11.897605 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:11.897440    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 15 21:30:12.902030 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:12.901999    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:30:12.903124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:12.903057    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:30:29.015308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:29.015271    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"]
Sep 15 21:30:43.466674 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.466590    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7445-41660f"]
Sep 15 21:30:43.470173 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.470151    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:43.472986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.472960    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mbx7k\""
Sep 15 21:30:43.476033 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.476009    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-41660f"]
Sep 15 21:30:43.580752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.580719    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzzf\" (UniqueName: \"kubernetes.io/projected/2f6b8427-8b8e-428b-b6a3-47de5adc179c-kube-api-access-xbzzf\") pod \"test-curl-7445-41660f\" (UID: \"2f6b8427-8b8e-428b-b6a3-47de5adc179c\") " pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:43.681689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.681644    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzzf\" (UniqueName: \"kubernetes.io/projected/2f6b8427-8b8e-428b-b6a3-47de5adc179c-kube-api-access-xbzzf\") pod \"test-curl-7445-41660f\" (UID: \"2f6b8427-8b8e-428b-b6a3-47de5adc179c\") " pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:43.689661 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.689626    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzzf\" (UniqueName: \"kubernetes.io/projected/2f6b8427-8b8e-428b-b6a3-47de5adc179c-kube-api-access-xbzzf\") pod \"test-curl-7445-41660f\" (UID: \"2f6b8427-8b8e-428b-b6a3-47de5adc179c\") " pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:43.782480 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:43.782387    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:44.035683 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.035561    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989" podUID="f9a5f552-fc84-45af-9cd6-7957736e9dc5" containerName="main" containerID="cri-o://602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add" gracePeriod=45
Sep 15 21:30:44.057541 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.057505    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-41660f"]
Sep 15 21:30:44.106291 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:44.106253    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f6b8427_8b8e_428b_b6a3_47de5adc179c.slice/crio-54afbd51a4912f6d5a749bd126b0dde6fa4ce13af62b885c9aa84b6019a9fe28 WatchSource:0}: Error finding container 54afbd51a4912f6d5a749bd126b0dde6fa4ce13af62b885c9aa84b6019a9fe28: Status 404 returned error can't find the container with id 54afbd51a4912f6d5a749bd126b0dde6fa4ce13af62b885c9aa84b6019a9fe28
Sep 15 21:30:44.289420 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.289352    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:30:44.388095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388066    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tmp-dir\") pod \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") "
Sep 15 21:30:44.388095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388099    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-model-cache\") pod \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") "
Sep 15 21:30:44.388376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388117    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-dshm\") pod \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") "
Sep 15 21:30:44.388376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388237    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-home\") pod \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") "
Sep 15 21:30:44.388376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388298    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tls-certs\") pod \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") "
Sep 15 21:30:44.388376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388323    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgj2x\" (UniqueName: \"kubernetes.io/projected/f9a5f552-fc84-45af-9cd6-7957736e9dc5-kube-api-access-kgj2x\") pod \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\" (UID: \"f9a5f552-fc84-45af-9cd6-7957736e9dc5\") "
Sep 15 21:30:44.388592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388398    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9a5f552-fc84-45af-9cd6-7957736e9dc5" (UID: "f9a5f552-fc84-45af-9cd6-7957736e9dc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:44.388592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9a5f552-fc84-45af-9cd6-7957736e9dc5" (UID: "f9a5f552-fc84-45af-9cd6-7957736e9dc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:44.388592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388533    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-home" (OuterVolumeSpecName: "home") pod "f9a5f552-fc84-45af-9cd6-7957736e9dc5" (UID: "f9a5f552-fc84-45af-9cd6-7957736e9dc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:44.388871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388835    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.389115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388891    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.389115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.388908    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.390704 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.390676    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-dshm" (OuterVolumeSpecName: "dshm") pod "f9a5f552-fc84-45af-9cd6-7957736e9dc5" (UID: "f9a5f552-fc84-45af-9cd6-7957736e9dc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:30:44.390704 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.390683    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9a5f552-fc84-45af-9cd6-7957736e9dc5" (UID: "f9a5f552-fc84-45af-9cd6-7957736e9dc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:44.390862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.390779    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a5f552-fc84-45af-9cd6-7957736e9dc5-kube-api-access-kgj2x" (OuterVolumeSpecName: "kube-api-access-kgj2x") pod "f9a5f552-fc84-45af-9cd6-7957736e9dc5" (UID: "f9a5f552-fc84-45af-9cd6-7957736e9dc5"). InnerVolumeSpecName "kube-api-access-kgj2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:44.489920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.489885    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a5f552-fc84-45af-9cd6-7957736e9dc5-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.489920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.489913    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a5f552-fc84-45af-9cd6-7957736e9dc5-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.489920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.489925    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgj2x\" (UniqueName: \"kubernetes.io/projected/f9a5f552-fc84-45af-9cd6-7957736e9dc5-kube-api-access-kgj2x\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:44.905471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.905431    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-41660f" event={"ID":"2f6b8427-8b8e-428b-b6a3-47de5adc179c","Type":"ContainerStarted","Data":"08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f"}
Sep 15 21:30:44.905471 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.905478    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-41660f" event={"ID":"2f6b8427-8b8e-428b-b6a3-47de5adc179c","Type":"ContainerStarted","Data":"54afbd51a4912f6d5a749bd126b0dde6fa4ce13af62b885c9aa84b6019a9fe28"}
Sep 15 21:30:44.906926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.906898    2551 generic.go:356] "Generic (PLEG): container finished" podID="f9a5f552-fc84-45af-9cd6-7957736e9dc5" containerID="602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add" exitCode=0
Sep 15 21:30:44.907059 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.906930    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"
Sep 15 21:30:44.907059 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.906972    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989" event={"ID":"f9a5f552-fc84-45af-9cd6-7957736e9dc5","Type":"ContainerDied","Data":"602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add"}
Sep 15 21:30:44.907059 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.907007    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989" event={"ID":"f9a5f552-fc84-45af-9cd6-7957736e9dc5","Type":"ContainerDied","Data":"8ecb5f5207079c6e4bda82fcfd0a893e47c43c4339e72b10a367db593214b067"}
Sep 15 21:30:44.907059 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.907029    2551 scope.go:117] "RemoveContainer" containerID="602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add"
Sep 15 21:30:44.917879 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.917860    2551 scope.go:117] "RemoveContainer" containerID="602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add"
Sep 15 21:30:44.918228 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:44.918206    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add\": container with ID starting with 602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add not found: ID does not exist" containerID="602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add"
Sep 15 21:30:44.918314 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.918235    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add"} err="failed to get container status \"602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add\": rpc error: code = NotFound desc = could not find container \"602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add\": container with ID starting with 602a09cee3b853cd1be38e6eee9759011e61457351c93de9a161c3121eab3add not found: ID does not exist"
Sep 15 21:30:44.921129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.921084    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7445-41660f" podStartSLOduration=1.241706763 podStartE2EDuration="1.92106723s" totalImagesPullingTime="679.360467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:44.919871242 +0000 UTC m=+1343.721740186" watchObservedRunningTime="2026-09-15 21:30:44.92106723 +0000 UTC m=+1343.722936169"
Sep 15 21:30:44.933116 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.933088    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"]
Sep 15 21:30:44.937315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:44.937292    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-70880c/test-model-70880c-kserve-689967bc69-b4989"]
Sep 15 21:30:45.526313 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.526147    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7445-41660f"]
Sep 15 21:30:45.530936 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.530887    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7445-41660f"]
Sep 15 21:30:45.649892 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.649855    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7445-b5293f"]
Sep 15 21:30:45.650286 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.650272    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a5f552-fc84-45af-9cd6-7957736e9dc5" containerName="main"
Sep 15 21:30:45.650336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.650288    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a5f552-fc84-45af-9cd6-7957736e9dc5" containerName="main"
Sep 15 21:30:45.650336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.650298    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f6b8427-8b8e-428b-b6a3-47de5adc179c" containerName="test-curl-7445-41660f"
Sep 15 21:30:45.650336 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.650304    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f6b8427-8b8e-428b-b6a3-47de5adc179c" containerName="test-curl-7445-41660f"
Sep 15 21:30:45.650435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.650362    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f6b8427-8b8e-428b-b6a3-47de5adc179c" containerName="test-curl-7445-41660f"
Sep 15 21:30:45.650435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.650371    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a5f552-fc84-45af-9cd6-7957736e9dc5" containerName="main"
Sep 15 21:30:45.652515 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.652499    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:45.659437 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.659411    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-b5293f"]
Sep 15 21:30:45.702756 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.702718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnfv\" (UniqueName: \"kubernetes.io/projected/68f054c1-4c45-41a3-b3d3-a8394238d82a-kube-api-access-nfnfv\") pod \"test-curl-7445-b5293f\" (UID: \"68f054c1-4c45-41a3-b3d3-a8394238d82a\") " pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:45.768809 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.768769    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a5f552-fc84-45af-9cd6-7957736e9dc5" path="/var/lib/kubelet/pods/f9a5f552-fc84-45af-9cd6-7957736e9dc5/volumes"
Sep 15 21:30:45.804450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.804361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnfv\" (UniqueName: \"kubernetes.io/projected/68f054c1-4c45-41a3-b3d3-a8394238d82a-kube-api-access-nfnfv\") pod \"test-curl-7445-b5293f\" (UID: \"68f054c1-4c45-41a3-b3d3-a8394238d82a\") " pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:45.812808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.812778    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnfv\" (UniqueName: \"kubernetes.io/projected/68f054c1-4c45-41a3-b3d3-a8394238d82a-kube-api-access-nfnfv\") pod \"test-curl-7445-b5293f\" (UID: \"68f054c1-4c45-41a3-b3d3-a8394238d82a\") " pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:45.913865 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.913831    2551 generic.go:356] "Generic (PLEG): container finished" podID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerID="e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0" exitCode=0
Sep 15 21:30:45.914068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.913899    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" event={"ID":"1b7bdbac-7830-4314-a910-ddcd3bb9c73f","Type":"ContainerDied","Data":"e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"}
Sep 15 21:30:45.914423 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.914405    2551 scope.go:117] "RemoveContainer" containerID="e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"
Sep 15 21:30:45.915307 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:45.915063    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7445-41660f" container="test-curl-7445-41660f"
Sep 15 21:30:45.915635 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.915604    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:45.915688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.915675    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:45.915729 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.915710    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:45.963565 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:45.963537    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:46.243666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:46.243626    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-b5293f"]
Sep 15 21:30:46.244534 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:46.244496    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68f054c1_4c45_41a3_b3d3_a8394238d82a.slice/crio-5c0ae6058062558cfbb6fbdc9c684226a621d82305666095801e74b6c415cd4f WatchSource:0}: Error finding container 5c0ae6058062558cfbb6fbdc9c684226a621d82305666095801e74b6c415cd4f: Status 404 returned error can't find the container with id 5c0ae6058062558cfbb6fbdc9c684226a621d82305666095801e74b6c415cd4f
Sep 15 21:30:46.921781 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:46.921744    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-b5293f" event={"ID":"68f054c1-4c45-41a3-b3d3-a8394238d82a","Type":"ContainerStarted","Data":"7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6"}
Sep 15 21:30:46.922274 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:46.921795    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-b5293f" event={"ID":"68f054c1-4c45-41a3-b3d3-a8394238d82a","Type":"ContainerStarted","Data":"5c0ae6058062558cfbb6fbdc9c684226a621d82305666095801e74b6c415cd4f"}
Sep 15 21:30:46.923938 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:46.923912    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" event={"ID":"1b7bdbac-7830-4314-a910-ddcd3bb9c73f","Type":"ContainerStarted","Data":"17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f"}
Sep 15 21:30:46.924033 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:46.923961    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7445-41660f" podUID="2f6b8427-8b8e-428b-b6a3-47de5adc179c" containerName="test-curl-7445-41660f" containerID="cri-o://08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f" gracePeriod=2
Sep 15 21:30:46.937684 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:46.937642    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7445-b5293f" podStartSLOduration=1.9376275760000001 podStartE2EDuration="1.937627576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:46.935022317 +0000 UTC m=+1345.736891259" watchObservedRunningTime="2026-09-15 21:30:46.937627576 +0000 UTC m=+1345.739496519"
Sep 15 21:30:47.435663 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.435632    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:47.435935 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.435913    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:30:47.435983 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.435961    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:30:47.779814 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.779694    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7445-b5293f"]
Sep 15 21:30:47.783771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.783739    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7445-b5293f"]
Sep 15 21:30:47.895799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.895767    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7445-865b50"]
Sep 15 21:30:47.896178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.896164    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" containerName="test-curl-7445-b5293f"
Sep 15 21:30:47.896259 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.896179    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" containerName="test-curl-7445-b5293f"
Sep 15 21:30:47.896294 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.896270    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" containerName="test-curl-7445-b5293f"
Sep 15 21:30:47.898555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.898538    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:47.908057 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:47.908032    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-865b50"]
Sep 15 21:30:47.929930 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:47.929902    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7445-b5293f" container="test-curl-7445-b5293f"
Sep 15 21:30:48.030998 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.030900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcbs\" (UniqueName: \"kubernetes.io/projected/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5-kube-api-access-btcbs\") pod \"test-curl-7445-865b50\" (UID: \"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5\") " pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:48.131885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.131855    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcbs\" (UniqueName: \"kubernetes.io/projected/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5-kube-api-access-btcbs\") pod \"test-curl-7445-865b50\" (UID: \"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5\") " pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:48.142283 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.142251    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcbs\" (UniqueName: \"kubernetes.io/projected/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5-kube-api-access-btcbs\") pod \"test-curl-7445-865b50\" (UID: \"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5\") " pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:48.208795 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.208762    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:48.435659 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.435616    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:30:48.435862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.435679    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:30:48.491499 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.488386    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-865b50"]
Sep 15 21:30:48.491831 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:48.491780    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f5c9fb2_765d_488d_bd1d_19a83cd56bf5.slice/crio-601a2ef848d95821577566534aa4b6d608a47c02831121591e6380870b358769 WatchSource:0}: Error finding container 601a2ef848d95821577566534aa4b6d608a47c02831121591e6380870b358769: Status 404 returned error can't find the container with id 601a2ef848d95821577566534aa4b6d608a47c02831121591e6380870b358769
Sep 15 21:30:48.663776 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.663671    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"]
Sep 15 21:30:48.663959 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.663922    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" podUID="c931a649-057e-4908-aefa-f799e3a56de9" containerName="maas-api" containerID="cri-o://19e722471a819a89f7a36818c176ed585bc025708659aaf9516773b4633233a6" gracePeriod=30
Sep 15 21:30:48.938614 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.938541    2551 generic.go:356] "Generic (PLEG): container finished" podID="c931a649-057e-4908-aefa-f799e3a56de9" containerID="19e722471a819a89f7a36818c176ed585bc025708659aaf9516773b4633233a6" exitCode=0
Sep 15 21:30:48.938946 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.938606    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" event={"ID":"c931a649-057e-4908-aefa-f799e3a56de9","Type":"ContainerDied","Data":"19e722471a819a89f7a36818c176ed585bc025708659aaf9516773b4633233a6"}
Sep 15 21:30:48.940277 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.940248    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-865b50" event={"ID":"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5","Type":"ContainerStarted","Data":"142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3"}
Sep 15 21:30:48.940392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.940282    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-865b50" event={"ID":"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5","Type":"ContainerStarted","Data":"601a2ef848d95821577566534aa4b6d608a47c02831121591e6380870b358769"}
Sep 15 21:30:48.940392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.940345    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7445-b5293f" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" containerName="test-curl-7445-b5293f" containerID="cri-o://7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6" gracePeriod=2
Sep 15 21:30:48.957507 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:48.957476    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7445-b5293f\" is forbidden: User \"system:node:ip-10-0-141-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-188.ec2.internal' and this object" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:48.958217 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:48.958162    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7445-865b50" podStartSLOduration=1.958147675 podStartE2EDuration="1.958147675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:48.955455231 +0000 UTC m=+1347.757324178" watchObservedRunningTime="2026-09-15 21:30:48.958147675 +0000 UTC m=+1347.760016617"
Sep 15 21:30:48.959386 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:48.959362    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7445-b5293f\" is forbidden: User \"system:node:ip-10-0-141-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-188.ec2.internal' and this object" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:49.041156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.040854    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:30:49.101299 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:49.098744    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7445-b5293f\" is forbidden: User \"system:node:ip-10-0-141-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-188.ec2.internal' and this object" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:49.142140 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.142106    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-metrics-certs\") pod \"c931a649-057e-4908-aefa-f799e3a56de9\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") "
Sep 15 21:30:49.142476 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.142397    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls\") pod \"c931a649-057e-4908-aefa-f799e3a56de9\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") "
Sep 15 21:30:49.142476 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.142430    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931a649-057e-4908-aefa-f799e3a56de9-openshift-service-ca\") pod \"c931a649-057e-4908-aefa-f799e3a56de9\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") "
Sep 15 21:30:49.142802 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.142746    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr9js\" (UniqueName: \"kubernetes.io/projected/c931a649-057e-4908-aefa-f799e3a56de9-kube-api-access-nr9js\") pod \"c931a649-057e-4908-aefa-f799e3a56de9\" (UID: \"c931a649-057e-4908-aefa-f799e3a56de9\") "
Sep 15 21:30:49.142899 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.142805    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c931a649-057e-4908-aefa-f799e3a56de9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c931a649-057e-4908-aefa-f799e3a56de9" (UID: "c931a649-057e-4908-aefa-f799e3a56de9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:49.143401 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.143378    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c931a649-057e-4908-aefa-f799e3a56de9-openshift-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.145115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.145094    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c931a649-057e-4908-aefa-f799e3a56de9" (UID: "c931a649-057e-4908-aefa-f799e3a56de9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:49.145374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.145344    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c931a649-057e-4908-aefa-f799e3a56de9" (UID: "c931a649-057e-4908-aefa-f799e3a56de9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:49.145772 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.145747    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c931a649-057e-4908-aefa-f799e3a56de9-kube-api-access-nr9js" (OuterVolumeSpecName: "kube-api-access-nr9js") pod "c931a649-057e-4908-aefa-f799e3a56de9" (UID: "c931a649-057e-4908-aefa-f799e3a56de9"). InnerVolumeSpecName "kube-api-access-nr9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:49.225688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.225666    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:49.227596 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:49.227568    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7445-b5293f\" is forbidden: User \"system:node:ip-10-0-141-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-188.ec2.internal' and this object" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:49.244890 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.244866    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9js\" (UniqueName: \"kubernetes.io/projected/c931a649-057e-4908-aefa-f799e3a56de9-kube-api-access-nr9js\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.244890 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.244892    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-metrics-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.245009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.244901    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c931a649-057e-4908-aefa-f799e3a56de9-maas-api-tls\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.345498 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.345469    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbzzf\" (UniqueName: \"kubernetes.io/projected/2f6b8427-8b8e-428b-b6a3-47de5adc179c-kube-api-access-xbzzf\") pod \"2f6b8427-8b8e-428b-b6a3-47de5adc179c\" (UID: \"2f6b8427-8b8e-428b-b6a3-47de5adc179c\") "
Sep 15 21:30:49.347602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.347576    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6b8427-8b8e-428b-b6a3-47de5adc179c-kube-api-access-xbzzf" (OuterVolumeSpecName: "kube-api-access-xbzzf") pod "2f6b8427-8b8e-428b-b6a3-47de5adc179c" (UID: "2f6b8427-8b8e-428b-b6a3-47de5adc179c"). InnerVolumeSpecName "kube-api-access-xbzzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:49.435537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.435506    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:30:49.435714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.435559    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:30:49.447075 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.447048    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzzf\" (UniqueName: \"kubernetes.io/projected/2f6b8427-8b8e-428b-b6a3-47de5adc179c-kube-api-access-xbzzf\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:49.742028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.741992    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7445-865b50"]
Sep 15 21:30:49.746493 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.746463    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7445-865b50"]
Sep 15 21:30:49.795817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.794336    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f6b8427-8b8e-428b-b6a3-47de5adc179c" path="/var/lib/kubelet/pods/2f6b8427-8b8e-428b-b6a3-47de5adc179c/volumes"
Sep 15 21:30:49.855360 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855329    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7445-aa1964"]
Sep 15 21:30:49.855786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855770    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" containerName="test-curl-7445-865b50"
Sep 15 21:30:49.855882 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855788    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" containerName="test-curl-7445-865b50"
Sep 15 21:30:49.855882 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855822    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c931a649-057e-4908-aefa-f799e3a56de9" containerName="maas-api"
Sep 15 21:30:49.855882 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855831    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c931a649-057e-4908-aefa-f799e3a56de9" containerName="maas-api"
Sep 15 21:30:49.856051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855933    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" containerName="test-curl-7445-865b50"
Sep 15 21:30:49.856051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.855951    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c931a649-057e-4908-aefa-f799e3a56de9" containerName="maas-api"
Sep 15 21:30:49.858072 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.858052    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:49.863003 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.862979    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-aa1964"]
Sep 15 21:30:49.947004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.946967    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"
Sep 15 21:30:49.947482 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.947003    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h" event={"ID":"c931a649-057e-4908-aefa-f799e3a56de9","Type":"ContainerDied","Data":"4b1e7e4f3f098d59708415f598ba634cc8e336731e2b4ecd2ddd64be69a4be2a"}
Sep 15 21:30:49.947482 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.947069    2551 scope.go:117] "RemoveContainer" containerID="19e722471a819a89f7a36818c176ed585bc025708659aaf9516773b4633233a6"
Sep 15 21:30:49.948660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.948634    2551 generic.go:356] "Generic (PLEG): container finished" podID="2f6b8427-8b8e-428b-b6a3-47de5adc179c" containerID="08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f" exitCode=137
Sep 15 21:30:49.948762 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.948684    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-41660f"
Sep 15 21:30:49.949182 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:49.949130    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7445-865b50" container="test-curl-7445-865b50"
Sep 15 21:30:49.952584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.952556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9nj\" (UniqueName: \"kubernetes.io/projected/5b1b2717-b372-4e10-8621-b9cf65015497-kube-api-access-dj9nj\") pod \"test-curl-7445-aa1964\" (UID: \"5b1b2717-b372-4e10-8621-b9cf65015497\") " pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:49.956491 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.956475    2551 scope.go:117] "RemoveContainer" containerID="08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f"
Sep 15 21:30:49.964491 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.964472    2551 scope.go:117] "RemoveContainer" containerID="08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f"
Sep 15 21:30:49.964785 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:49.964766    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f\": container with ID starting with 08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f not found: ID does not exist" containerID="08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f"
Sep 15 21:30:49.964880 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.964794    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f"} err="failed to get container status \"08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f\": rpc error: code = NotFound desc = could not find container \"08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f\": container with ID starting with 08022ec502aa6d41f4b50783a41fa6372df698660a62d683d7f246d58328622f not found: ID does not exist"
Sep 15 21:30:49.965469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.965448    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"]
Sep 15 21:30:49.995277 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:49.994692    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-70880c-6c9c974775-jz57h"]
Sep 15 21:30:50.053818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.053783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9nj\" (UniqueName: \"kubernetes.io/projected/5b1b2717-b372-4e10-8621-b9cf65015497-kube-api-access-dj9nj\") pod \"test-curl-7445-aa1964\" (UID: \"5b1b2717-b372-4e10-8621-b9cf65015497\") " pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:50.063128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.063101    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9nj\" (UniqueName: \"kubernetes.io/projected/5b1b2717-b372-4e10-8621-b9cf65015497-kube-api-access-dj9nj\") pod \"test-curl-7445-aa1964\" (UID: \"5b1b2717-b372-4e10-8621-b9cf65015497\") " pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:50.188259 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.188218    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:50.435671 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.435636    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:30:50.435854 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.435700    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:30:50.456784 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.456763    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-aa1964"]
Sep 15 21:30:50.458840 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:50.458816    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b1b2717_b372_4e10_8621_b9cf65015497.slice/crio-3d33618e9167cfc76dedccd5df0b4f0011a985c2981e0398fee67eb9fb8e96a1 WatchSource:0}: Error finding container 3d33618e9167cfc76dedccd5df0b4f0011a985c2981e0398fee67eb9fb8e96a1: Status 404 returned error can't find the container with id 3d33618e9167cfc76dedccd5df0b4f0011a985c2981e0398fee67eb9fb8e96a1
Sep 15 21:30:50.957689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.957660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-aa1964" event={"ID":"5b1b2717-b372-4e10-8621-b9cf65015497","Type":"ContainerStarted","Data":"d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a"}
Sep 15 21:30:50.957689 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.957699    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-aa1964" event={"ID":"5b1b2717-b372-4e10-8621-b9cf65015497","Type":"ContainerStarted","Data":"3d33618e9167cfc76dedccd5df0b4f0011a985c2981e0398fee67eb9fb8e96a1"}
Sep 15 21:30:50.958765 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:50.958727    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7445-865b50" podUID="2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" containerName="test-curl-7445-865b50" containerID="cri-o://142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3" gracePeriod=2
Sep 15 21:30:51.137518 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.137494    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:51.266755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.266725    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfnfv\" (UniqueName: \"kubernetes.io/projected/68f054c1-4c45-41a3-b3d3-a8394238d82a-kube-api-access-nfnfv\") pod \"68f054c1-4c45-41a3-b3d3-a8394238d82a\" (UID: \"68f054c1-4c45-41a3-b3d3-a8394238d82a\") "
Sep 15 21:30:51.297623 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.297583    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f054c1-4c45-41a3-b3d3-a8394238d82a-kube-api-access-nfnfv" (OuterVolumeSpecName: "kube-api-access-nfnfv") pod "68f054c1-4c45-41a3-b3d3-a8394238d82a" (UID: "68f054c1-4c45-41a3-b3d3-a8394238d82a"). InnerVolumeSpecName "kube-api-access-nfnfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:51.384216 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.380847    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnfv\" (UniqueName: \"kubernetes.io/projected/68f054c1-4c45-41a3-b3d3-a8394238d82a-kube-api-access-nfnfv\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:51.435834 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.435801    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:30:51.436011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.435860    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 21:30:51.598474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.598354    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7445-aa1964" podStartSLOduration=2.598337141 podStartE2EDuration="2.598337141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:50.982154879 +0000 UTC m=+1349.784023825" watchObservedRunningTime="2026-09-15 21:30:51.598337141 +0000 UTC m=+1350.400206084"
Sep 15 21:30:51.600275 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.600256    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7445-aa1964"]
Sep 15 21:30:51.607668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.607639    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7445-aa1964"]
Sep 15 21:30:51.779281 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.779244    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f054c1-4c45-41a3-b3d3-a8394238d82a" path="/var/lib/kubelet/pods/68f054c1-4c45-41a3-b3d3-a8394238d82a/volumes"
Sep 15 21:30:51.779883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.779858    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c931a649-057e-4908-aefa-f799e3a56de9" path="/var/lib/kubelet/pods/c931a649-057e-4908-aefa-f799e3a56de9/volumes"
Sep 15 21:30:51.966247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.966210    2551 generic.go:356] "Generic (PLEG): container finished" podID="68f054c1-4c45-41a3-b3d3-a8394238d82a" containerID="7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6" exitCode=137
Sep 15 21:30:51.966727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.966256    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-b5293f"
Sep 15 21:30:51.966727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.966313    2551 scope.go:117] "RemoveContainer" containerID="7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6"
Sep 15 21:30:51.966727 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:51.966541    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7445-aa1964" container="test-curl-7445-aa1964"
Sep 15 21:30:51.976926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.976908    2551 scope.go:117] "RemoveContainer" containerID="7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6"
Sep 15 21:30:51.977312 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:51.977294    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6\": container with ID starting with 7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6 not found: ID does not exist" containerID="7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6"
Sep 15 21:30:51.977361 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:51.977322    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6"} err="failed to get container status \"7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6\": rpc error: code = NotFound desc = could not find container \"7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6\": container with ID starting with 7a0d677ab9c4482a3f3f15bd778a08ed608cd5dd5fd309d9482f24d6f6dfe5b6 not found: ID does not exist"
Sep 15 21:30:52.440029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.439993    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:52.440437 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.440421    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:52.440964 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.440944    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:30:52.470667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.470634    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"]
Sep 15 21:30:52.471058 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.471043    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1b2717-b372-4e10-8621-b9cf65015497" containerName="test-curl-7445-aa1964"
Sep 15 21:30:52.471142 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.471061    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1b2717-b372-4e10-8621-b9cf65015497" containerName="test-curl-7445-aa1964"
Sep 15 21:30:52.471142 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.471131    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1b2717-b372-4e10-8621-b9cf65015497" containerName="test-curl-7445-aa1964"
Sep 15 21:30:52.474282 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.474258    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.476395 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.476377    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4e476d59-openshift-default-dockercfg-jksn4\""
Sep 15 21:30:52.486165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.486143    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"]
Sep 15 21:30:52.599534 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-data\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.599705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599557    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-credential-socket\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.599705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599575    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-token\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.599705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599625    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlksh\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-kube-api-access-vlksh\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.599809 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-podinfo\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.599809 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599772    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-certs\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.599997 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.599973    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istiod-ca-cert\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.600079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.600058    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-envoy\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.600147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.600114    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-socket\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701291    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-data\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701338    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-credential-socket\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701356    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-token\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701380    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlksh\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-kube-api-access-vlksh\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701420    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-podinfo\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701439    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-certs\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701484    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istiod-ca-cert\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701541    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-envoy\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.701754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701591    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-socket\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.702028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701776    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-credential-socket\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.702028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701935    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-data\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.702028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.701954    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-socket\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.702028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.702015    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-certs\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.702374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.702353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istiod-ca-cert\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.704001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.703982    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-envoy\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.704084 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.704033    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-podinfo\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.708944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.708924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-token\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.709037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.709021    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlksh\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-kube-api-access-vlksh\") pod \"e2e-mt-4e476d59-openshift-default-669cf6f668-mws26\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") " pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.784898 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.784870    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:52.974988 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:52.974939    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7445-aa1964" podUID="5b1b2717-b372-4e10-8621-b9cf65015497" containerName="test-curl-7445-aa1964" containerID="cri-o://d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a" gracePeriod=2
Sep 15 21:30:53.062848 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.062819    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:53.072287 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.072262    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"]
Sep 15 21:30:53.073922 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:30:53.073896    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b90fcf5_05dc_4164_b3fc_4342e91055d3.slice/crio-b2ef9179ad0872ff9facd53b6ac5ea60cfeba269468eaff67946aad0630fdb5e WatchSource:0}: Error finding container b2ef9179ad0872ff9facd53b6ac5ea60cfeba269468eaff67946aad0630fdb5e: Status 404 returned error can't find the container with id b2ef9179ad0872ff9facd53b6ac5ea60cfeba269468eaff67946aad0630fdb5e
Sep 15 21:30:53.076315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.076272    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:53.076439 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.076360    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:53.076439 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.076407    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:30:53.207041 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.207001    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btcbs\" (UniqueName: \"kubernetes.io/projected/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5-kube-api-access-btcbs\") pod \"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5\" (UID: \"2f5c9fb2-765d-488d-bd1d-19a83cd56bf5\") "
Sep 15 21:30:53.209485 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.209455    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5-kube-api-access-btcbs" (OuterVolumeSpecName: "kube-api-access-btcbs") pod "2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" (UID: "2f5c9fb2-765d-488d-bd1d-19a83cd56bf5"). InnerVolumeSpecName "kube-api-access-btcbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:53.308729 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.308640    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcbs\" (UniqueName: \"kubernetes.io/projected/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5-kube-api-access-btcbs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:53.769121 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.769076    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" path="/var/lib/kubelet/pods/2f5c9fb2-765d-488d-bd1d-19a83cd56bf5/volumes"
Sep 15 21:30:53.981939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.981839    2551 generic.go:356] "Generic (PLEG): container finished" podID="2f5c9fb2-765d-488d-bd1d-19a83cd56bf5" containerID="142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3" exitCode=137
Sep 15 21:30:53.981939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.981895    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-865b50"
Sep 15 21:30:53.982447 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.981942    2551 scope.go:117] "RemoveContainer" containerID="142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3"
Sep 15 21:30:53.984068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.984035    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" event={"ID":"9b90fcf5-05dc-4164-b3fc-4342e91055d3","Type":"ContainerStarted","Data":"32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"}
Sep 15 21:30:53.984218 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.984179    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" event={"ID":"9b90fcf5-05dc-4164-b3fc-4342e91055d3","Type":"ContainerStarted","Data":"b2ef9179ad0872ff9facd53b6ac5ea60cfeba269468eaff67946aad0630fdb5e"}
Sep 15 21:30:53.990990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.990976    2551 scope.go:117] "RemoveContainer" containerID="142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3"
Sep 15 21:30:53.991340 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:53.991321    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3\": container with ID starting with 142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3 not found: ID does not exist" containerID="142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3"
Sep 15 21:30:53.991417 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:53.991350    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3"} err="failed to get container status \"142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3\": rpc error: code = NotFound desc = could not find container \"142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3\": container with ID starting with 142da4a6dc7ea74304d5f43b4e75e34ec06aafcfb5b79649bdf38c8f88f883e3 not found: ID does not exist"
Sep 15 21:30:54.006803 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:54.006762    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podStartSLOduration=2.006749135 podStartE2EDuration="2.006749135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:54.005264052 +0000 UTC m=+1352.807133025" watchObservedRunningTime="2026-09-15 21:30:54.006749135 +0000 UTC m=+1352.808618078"
Sep 15 21:30:54.785594 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:54.785558    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:54.791355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:54.791328    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:54.991980 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:54.991949    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:54.993068 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:54.993048    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:30:55.160465 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.160444    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:55.232414 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.232353    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj9nj\" (UniqueName: \"kubernetes.io/projected/5b1b2717-b372-4e10-8621-b9cf65015497-kube-api-access-dj9nj\") pod \"5b1b2717-b372-4e10-8621-b9cf65015497\" (UID: \"5b1b2717-b372-4e10-8621-b9cf65015497\") "
Sep 15 21:30:55.235926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.235891    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1b2717-b372-4e10-8621-b9cf65015497-kube-api-access-dj9nj" (OuterVolumeSpecName: "kube-api-access-dj9nj") pod "5b1b2717-b372-4e10-8621-b9cf65015497" (UID: "5b1b2717-b372-4e10-8621-b9cf65015497"). InnerVolumeSpecName "kube-api-access-dj9nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:55.252325 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.252298    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"]
Sep 15 21:30:55.252510 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.252489    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" containerID="cri-o://21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98" gracePeriod=30
Sep 15 21:30:55.334340 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.334254    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj9nj\" (UniqueName: \"kubernetes.io/projected/5b1b2717-b372-4e10-8621-b9cf65015497-kube-api-access-dj9nj\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:55.636385 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.636307    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"]
Sep 15 21:30:55.638923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.638909    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:55.641060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.641036    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:30:55.647528 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.647503    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"]
Sep 15 21:30:55.739304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.739267    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b8c6f-8e2f-40ea-8152-e0677d1de77c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-779ace-5958c\" (UID: \"293b8c6f-8e2f-40ea-8152-e0677d1de77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:55.769393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.769363    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1b2717-b372-4e10-8621-b9cf65015497" path="/var/lib/kubelet/pods/5b1b2717-b372-4e10-8621-b9cf65015497/volumes"
Sep 15 21:30:55.840429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.840390    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b8c6f-8e2f-40ea-8152-e0677d1de77c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-779ace-5958c\" (UID: \"293b8c6f-8e2f-40ea-8152-e0677d1de77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:55.840987 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.840968    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b8c6f-8e2f-40ea-8152-e0677d1de77c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-779ace-5958c\" (UID: \"293b8c6f-8e2f-40ea-8152-e0677d1de77c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:55.950747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:55.950720    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:56.001798 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:56.001677    2551 generic.go:356] "Generic (PLEG): container finished" podID="5b1b2717-b372-4e10-8621-b9cf65015497" containerID="d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a" exitCode=137
Sep 15 21:30:56.002261 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:56.002239    2551 scope.go:117] "RemoveContainer" containerID="d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a"
Sep 15 21:30:56.002416 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:56.002399    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-aa1964"
Sep 15 21:30:56.013772 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:56.013629    2551 scope.go:117] "RemoveContainer" containerID="d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a"
Sep 15 21:30:56.014491 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:30:56.014463    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a\": container with ID starting with d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a not found: ID does not exist" containerID="d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a"
Sep 15 21:30:56.014617 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:56.014506    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a"} err="failed to get container status \"d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a\": rpc error: code = NotFound desc = could not find container \"d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a\": container with ID starting with d0bc8a322bba94487cd9821b9f153dae5fc9d5b1e37d66b3ea7d1e1a1a21978a not found: ID does not exist"
Sep 15 21:30:56.229691 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:56.229642    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"]
Sep 15 21:30:57.010577 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:57.010543    2551 generic.go:356] "Generic (PLEG): container finished" podID="293b8c6f-8e2f-40ea-8152-e0677d1de77c" containerID="b0ca3c92ed0dd0eb36a8023f91b8603e6fe5578e6069106ae02546fc89788dc9" exitCode=0
Sep 15 21:30:57.010939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:57.010623    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c" event={"ID":"293b8c6f-8e2f-40ea-8152-e0677d1de77c","Type":"ContainerDied","Data":"b0ca3c92ed0dd0eb36a8023f91b8603e6fe5578e6069106ae02546fc89788dc9"}
Sep 15 21:30:57.010939 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:57.010704    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c" event={"ID":"293b8c6f-8e2f-40ea-8152-e0677d1de77c","Type":"ContainerStarted","Data":"695dc65d8fd13bc9de073f74400912b4d2f4d8da720cf45b783d73b69be07a5b"}
Sep 15 21:30:58.159803 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:58.159780    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:58.272241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:58.272123    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b8c6f-8e2f-40ea-8152-e0677d1de77c-maas-api-service-ca\") pod \"293b8c6f-8e2f-40ea-8152-e0677d1de77c\" (UID: \"293b8c6f-8e2f-40ea-8152-e0677d1de77c\") "
Sep 15 21:30:58.272526 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:58.272504    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293b8c6f-8e2f-40ea-8152-e0677d1de77c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "293b8c6f-8e2f-40ea-8152-e0677d1de77c" (UID: "293b8c6f-8e2f-40ea-8152-e0677d1de77c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:58.373657 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:58.373615    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b8c6f-8e2f-40ea-8152-e0677d1de77c-maas-api-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:30:59.022351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:59.022324    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"
Sep 15 21:30:59.022525 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:59.022366    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c" event={"ID":"293b8c6f-8e2f-40ea-8152-e0677d1de77c","Type":"ContainerDied","Data":"695dc65d8fd13bc9de073f74400912b4d2f4d8da720cf45b783d73b69be07a5b"}
Sep 15 21:30:59.022525 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:59.022410    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695dc65d8fd13bc9de073f74400912b4d2f4d8da720cf45b783d73b69be07a5b"
Sep 15 21:30:59.607920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:59.607884    2551 patch_prober.go:28] interesting pod/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:30:59.608350 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:30:59.607943    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:31:00.511919 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.511897    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:31:00.699023 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.698988    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-podinfo\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699037    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-data\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699076    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istiod-ca-cert\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699155    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjrpt\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-kube-api-access-jjrpt\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699223    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-token\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699257    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-certs\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699307    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-envoy\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699351    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-socket\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699437    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-credential-socket\") pod \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\" (UID: \"2b6b50c6-518d-4090-87fb-38ea2fd328a0\") "
Sep 15 21:31:00.699537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699448    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699536    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699670    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699805    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699822    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699837    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699804    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.699951 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.699807    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.701581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.701552    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:00.701581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.701558    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:00.701846 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.701828    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:31:00.701907 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.701840    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-kube-api-access-jjrpt" (OuterVolumeSpecName: "kube-api-access-jjrpt") pod "2b6b50c6-518d-4090-87fb-38ea2fd328a0" (UID: "2b6b50c6-518d-4090-87fb-38ea2fd328a0"). InnerVolumeSpecName "kube-api-access-jjrpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:00.800825 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.800787    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.800825 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.800820    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.801005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.800833    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjrpt\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-kube-api-access-jjrpt\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.801005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.800847    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.801005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.800859    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.801005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.800869    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6b50c6-518d-4090-87fb-38ea2fd328a0-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:00.851055 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851023    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"]
Sep 15 21:31:00.851646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851618    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy"
Sep 15 21:31:00.851646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851641    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy"
Sep 15 21:31:00.851818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851682    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="293b8c6f-8e2f-40ea-8152-e0677d1de77c" containerName="revoke-keys"
Sep 15 21:31:00.851818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851692    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="293b8c6f-8e2f-40ea-8152-e0677d1de77c" containerName="revoke-keys"
Sep 15 21:31:00.851818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851785    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="293b8c6f-8e2f-40ea-8152-e0677d1de77c" containerName="revoke-keys"
Sep 15 21:31:00.851818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851796    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy"
Sep 15 21:31:00.851818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.851809    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy"
Sep 15 21:31:00.855263 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.855241    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:00.857584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.857562    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4e476d59-serving-cert\""
Sep 15 21:31:00.867513 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:00.867485    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"]
Sep 15 21:31:01.002694 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.002611    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbsl\" (UniqueName: \"kubernetes.io/projected/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-kube-api-access-2tbsl\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.002694 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.002652    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-maas-api-tls\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.002877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.002745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-metrics-certs\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.002933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.002907    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-openshift-service-ca\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.034834 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.034807    2551 generic.go:356] "Generic (PLEG): container finished" podID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerID="21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98" exitCode=0
Sep 15 21:31:01.035004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.034856    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" event={"ID":"2b6b50c6-518d-4090-87fb-38ea2fd328a0","Type":"ContainerDied","Data":"21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98"}
Sep 15 21:31:01.035004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.034862    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"
Sep 15 21:31:01.035004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.034898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c" event={"ID":"2b6b50c6-518d-4090-87fb-38ea2fd328a0","Type":"ContainerDied","Data":"4aac28d6e898eb009ecd860360d9e146087abf54db89925a85dfc0301a129938"}
Sep 15 21:31:01.035004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.034920    2551 scope.go:117] "RemoveContainer" containerID="21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98"
Sep 15 21:31:01.044592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.044572    2551 scope.go:117] "RemoveContainer" containerID="b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"
Sep 15 21:31:01.060086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.060056    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"]
Sep 15 21:31:01.061701 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.061679    2551 scope.go:117] "RemoveContainer" containerID="21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98"
Sep 15 21:31:01.062041 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:31:01.062022    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98\": container with ID starting with 21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98 not found: ID does not exist" containerID="21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98"
Sep 15 21:31:01.062120 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.062050    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98"} err="failed to get container status \"21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98\": rpc error: code = NotFound desc = could not find container \"21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98\": container with ID starting with 21d953d2d39d4439d3d46769b27ccef2f84de7b96b97d9ba63080c4c8c369a98 not found: ID does not exist"
Sep 15 21:31:01.062120 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.062069    2551 scope.go:117] "RemoveContainer" containerID="b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"
Sep 15 21:31:01.062436 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:31:01.062409    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c\": container with ID starting with b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c not found: ID does not exist" containerID="b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"
Sep 15 21:31:01.062505 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.062447    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c"} err="failed to get container status \"b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c\": rpc error: code = NotFound desc = could not find container \"b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c\": container with ID starting with b7c06e56e5c4a8b1436513095c8187f69e7613b593e23b2d5f057cfac1f5379c not found: ID does not exist"
Sep 15 21:31:01.063983 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.063967    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-70880c-openshift-default-59bfbbc698-4tl5c"]
Sep 15 21:31:01.104159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.104129    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-openshift-service-ca\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.104306 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.104217    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbsl\" (UniqueName: \"kubernetes.io/projected/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-kube-api-access-2tbsl\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.104306 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.104242    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-maas-api-tls\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.104381 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.104308    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-metrics-certs\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.104827 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.104809    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-openshift-service-ca\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.106902 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.106883    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-maas-api-tls\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.106963 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.106915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-metrics-certs\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.112040 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.112023    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbsl\" (UniqueName: \"kubernetes.io/projected/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-kube-api-access-2tbsl\") pod \"maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.166911 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.166890    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:01.439872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.439842    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"]
Sep 15 21:31:01.441748 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:31:01.441718    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543a10d3_ca8e_4f69_bbfe_aaa9bc90c286.slice/crio-709b61ca15b1e42f3efd450e67a1de2cb3be6529491785b0c20a0dfb97590353 WatchSource:0}: Error finding container 709b61ca15b1e42f3efd450e67a1de2cb3be6529491785b0c20a0dfb97590353: Status 404 returned error can't find the container with id 709b61ca15b1e42f3efd450e67a1de2cb3be6529491785b0c20a0dfb97590353
Sep 15 21:31:01.769728 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:01.769655    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" path="/var/lib/kubelet/pods/2b6b50c6-518d-4090-87fb-38ea2fd328a0/volumes"
Sep 15 21:31:02.041626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:02.041528    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k" event={"ID":"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286","Type":"ContainerStarted","Data":"6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8"}
Sep 15 21:31:02.041626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:02.041583    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k" event={"ID":"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286","Type":"ContainerStarted","Data":"709b61ca15b1e42f3efd450e67a1de2cb3be6529491785b0c20a0dfb97590353"}
Sep 15 21:31:02.041836 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:02.041691    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:02.067465 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:02.067409    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k" podStartSLOduration=1.637272838 podStartE2EDuration="2.067387509s" totalImagesPullingTime="430.114671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:02.064865544 +0000 UTC m=+1360.866734511" watchObservedRunningTime="2026-09-15 21:31:02.067387509 +0000 UTC m=+1360.869256453"
Sep 15 21:31:06.574807 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:06.574777    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"]
Sep 15 21:31:06.583753 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:06.583725    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-779ace-5958c"]
Sep 15 21:31:07.768771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:07.768739    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="293b8c6f-8e2f-40ea-8152-e0677d1de77c" path="/var/lib/kubelet/pods/293b8c6f-8e2f-40ea-8152-e0677d1de77c/volumes"
Sep 15 21:31:08.054307 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:08.054227    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:31:10.714392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.714359    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"]
Sep 15 21:31:10.714777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.714740    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy"
Sep 15 21:31:10.714777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.714751    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6b50c6-518d-4090-87fb-38ea2fd328a0" containerName="istio-proxy"
Sep 15 21:31:10.717477 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.717453    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.719954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.719930    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d130a8\"/\"openshift-service-ca.crt\""
Sep 15 21:31:10.720707 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.720682    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d130a8\"/\"kube-root-ca.crt\""
Sep 15 21:31:10.720756 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.720730    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d130a8\"/\"default-dockercfg-rqxnx\""
Sep 15 21:31:10.720807 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.720752    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d130a8\"/\"noten-model-d130a8-kserve-self-signed-certs\""
Sep 15 21:31:10.730976 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.730951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"]
Sep 15 21:31:10.813628 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.813594    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-dshm\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.813628 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.813632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tmp-dir\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.813830 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.813715    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tls-certs\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.813830 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.813773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-model-cache\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.813830 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.813809    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-home\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.813919 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.813838    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jnd\" (UniqueName: \"kubernetes.io/projected/38adf8c0-69c3-4bab-8aff-12aae31f8eae-kube-api-access-g9jnd\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915400 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tls-certs\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915426    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-model-cache\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915452    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-home\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jnd\" (UniqueName: \"kubernetes.io/projected/38adf8c0-69c3-4bab-8aff-12aae31f8eae-kube-api-access-g9jnd\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915530    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-dshm\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tmp-dir\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915910 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915885    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-model-cache\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915971 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915932    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-home\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.915971 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.915945    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tmp-dir\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.918016 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.917999    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-dshm\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.918177 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.918162    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tls-certs\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:10.924831 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:10.924807    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jnd\" (UniqueName: \"kubernetes.io/projected/38adf8c0-69c3-4bab-8aff-12aae31f8eae-kube-api-access-g9jnd\") pod \"noten-model-d130a8-kserve-67c9b76b87-v22wm\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") " pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:11.027570 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:11.027491    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:11.309001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:11.308977    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"]
Sep 15 21:31:11.311160 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:31:11.311134    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38adf8c0_69c3_4bab_8aff_12aae31f8eae.slice/crio-ac5c09b2e408576a34f9578e61e905fb60e460c27555452e7724e5cb501e962d WatchSource:0}: Error finding container ac5c09b2e408576a34f9578e61e905fb60e460c27555452e7724e5cb501e962d: Status 404 returned error can't find the container with id ac5c09b2e408576a34f9578e61e905fb60e460c27555452e7724e5cb501e962d
Sep 15 21:31:11.962022 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:11.961990    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"]
Sep 15 21:31:12.110969 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:12.110931    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm" event={"ID":"38adf8c0-69c3-4bab-8aff-12aae31f8eae","Type":"ContainerStarted","Data":"0616d5fd851f7d0e2b227e8e28ff201af95214a7d01f2152d444889c0fc43eae"}
Sep 15 21:31:12.111117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:12.110981    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm" event={"ID":"38adf8c0-69c3-4bab-8aff-12aae31f8eae","Type":"ContainerStarted","Data":"ac5c09b2e408576a34f9578e61e905fb60e460c27555452e7724e5cb501e962d"}
Sep 15 21:31:12.137144 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:12.137094    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm" podStartSLOduration=2.137078873 podStartE2EDuration="2.137078873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:12.135328402 +0000 UTC m=+1370.937197350" watchObservedRunningTime="2026-09-15 21:31:12.137078873 +0000 UTC m=+1370.938947854"
Sep 15 21:31:21.027897 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:21.027861    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:21.028472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:21.027917    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:21.040482 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:21.040454    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:21.197876 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:21.197843    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:31:26.982165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:26.982098    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw" podUID="ad28e70b-4637-4a69-843e-239870969716" containerName="main" containerID="cri-o://57289cea4f6249a4d2cbfd59c58537dc90cdb3b7590db41052f8c52e4436cbd7" gracePeriod=45
Sep 15 21:31:27.206142 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.206101    2551 generic.go:356] "Generic (PLEG): container finished" podID="ad28e70b-4637-4a69-843e-239870969716" containerID="57289cea4f6249a4d2cbfd59c58537dc90cdb3b7590db41052f8c52e4436cbd7" exitCode=0
Sep 15 21:31:27.206349 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.206144    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw" event={"ID":"ad28e70b-4637-4a69-843e-239870969716","Type":"ContainerDied","Data":"57289cea4f6249a4d2cbfd59c58537dc90cdb3b7590db41052f8c52e4436cbd7"}
Sep 15 21:31:27.237350 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.237284    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:31:27.391911 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.391879    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcmwg\" (UniqueName: \"kubernetes.io/projected/ad28e70b-4637-4a69-843e-239870969716-kube-api-access-gcmwg\") pod \"ad28e70b-4637-4a69-843e-239870969716\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") "
Sep 15 21:31:27.392115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.391973    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-tmp-dir\") pod \"ad28e70b-4637-4a69-843e-239870969716\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") "
Sep 15 21:31:27.392115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392012    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-dshm\") pod \"ad28e70b-4637-4a69-843e-239870969716\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") "
Sep 15 21:31:27.392115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392046    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-model-cache\") pod \"ad28e70b-4637-4a69-843e-239870969716\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") "
Sep 15 21:31:27.392436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392135    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad28e70b-4637-4a69-843e-239870969716-tls-certs\") pod \"ad28e70b-4637-4a69-843e-239870969716\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") "
Sep 15 21:31:27.392436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392166    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-home\") pod \"ad28e70b-4637-4a69-843e-239870969716\" (UID: \"ad28e70b-4637-4a69-843e-239870969716\") "
Sep 15 21:31:27.392436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392323    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad28e70b-4637-4a69-843e-239870969716" (UID: "ad28e70b-4637-4a69-843e-239870969716"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:27.392436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392339    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad28e70b-4637-4a69-843e-239870969716" (UID: "ad28e70b-4637-4a69-843e-239870969716"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:27.392653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392519    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-home" (OuterVolumeSpecName: "home") pod "ad28e70b-4637-4a69-843e-239870969716" (UID: "ad28e70b-4637-4a69-843e-239870969716"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:27.392702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392688    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:27.392748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392725    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:27.392748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.392739    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:27.395005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.394969    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-dshm" (OuterVolumeSpecName: "dshm") pod "ad28e70b-4637-4a69-843e-239870969716" (UID: "ad28e70b-4637-4a69-843e-239870969716"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:27.395125 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.395020    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad28e70b-4637-4a69-843e-239870969716-kube-api-access-gcmwg" (OuterVolumeSpecName: "kube-api-access-gcmwg") pod "ad28e70b-4637-4a69-843e-239870969716" (UID: "ad28e70b-4637-4a69-843e-239870969716"). InnerVolumeSpecName "kube-api-access-gcmwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:27.395125 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.395043    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad28e70b-4637-4a69-843e-239870969716-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad28e70b-4637-4a69-843e-239870969716" (UID: "ad28e70b-4637-4a69-843e-239870969716"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:27.493877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.493792    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcmwg\" (UniqueName: \"kubernetes.io/projected/ad28e70b-4637-4a69-843e-239870969716-kube-api-access-gcmwg\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:27.493877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.493821    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad28e70b-4637-4a69-843e-239870969716-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:27.493877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:27.493831    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad28e70b-4637-4a69-843e-239870969716-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:28.213994 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:28.213965    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"
Sep 15 21:31:28.214463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:28.213966    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw" event={"ID":"ad28e70b-4637-4a69-843e-239870969716","Type":"ContainerDied","Data":"2f79694d831e24dbca9425c83a5809c8db3d116e7e89985988be5749691f23d9"}
Sep 15 21:31:28.214463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:28.214091    2551 scope.go:117] "RemoveContainer" containerID="57289cea4f6249a4d2cbfd59c58537dc90cdb3b7590db41052f8c52e4436cbd7"
Sep 15 21:31:28.230037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:28.229999    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"]
Sep 15 21:31:28.236319 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:28.236293    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-779ace/test-model-779ace-kserve-74d4c599df-8srfw"]
Sep 15 21:31:29.775709 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:29.775674    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad28e70b-4637-4a69-843e-239870969716" path="/var/lib/kubelet/pods/ad28e70b-4637-4a69-843e-239870969716/volumes"
Sep 15 21:31:30.226239 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:30.226179    2551 generic.go:356] "Generic (PLEG): container finished" podID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerID="32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c" exitCode=0
Sep 15 21:31:30.226425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:30.226253    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" event={"ID":"9b90fcf5-05dc-4164-b3fc-4342e91055d3","Type":"ContainerDied","Data":"32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"}
Sep 15 21:31:30.226653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:30.226638    2551 scope.go:117] "RemoveContainer" containerID="32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"
Sep 15 21:31:30.227889 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:30.227860    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:31:30.227945 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:30.227931    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:31:30.227990 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:30.227963    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:31:31.233368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:31.233336    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" event={"ID":"9b90fcf5-05dc-4164-b3fc-4342e91055d3","Type":"ContainerStarted","Data":"5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc"}
Sep 15 21:31:31.785469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:31.785434    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:31:31.785736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:31.785715    2551 patch_prober.go:28] interesting pod/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:31:31.785787 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:31.785761    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:31:32.786154 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:32.786119    2551 patch_prober.go:28] interesting pod/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:31:32.786799 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:32.786751    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:31:33.785896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:33.785850    2551 patch_prober.go:28] interesting pod/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:31:33.786112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:33.785918    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:31:34.786369 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:34.786324    2551 patch_prober.go:28] interesting pod/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:31:34.786890 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:34.786395    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:31:35.785382 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:35.785338    2551 patch_prober.go:28] interesting pod/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:31:35.785603 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:35.785422    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:31:36.785386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:36.785347    2551 patch_prober.go:28] interesting pod/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:31:36.786012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:36.785973    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 15 21:31:37.819045 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:37.819015    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:31:37.819534 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:37.819425    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:31:37.820360 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:37.820340    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:31:45.456367 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.456332    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"]
Sep 15 21:31:45.456747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.456722    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad28e70b-4637-4a69-843e-239870969716" containerName="main"
Sep 15 21:31:45.456747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.456732    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad28e70b-4637-4a69-843e-239870969716" containerName="main"
Sep 15 21:31:45.456819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.456802    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad28e70b-4637-4a69-843e-239870969716" containerName="main"
Sep 15 21:31:45.459567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.459552    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.462327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.462301    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-625609\"/\"e2e-conflict-b-23fd2eb6-kserve-self-signed-certs\""
Sep 15 21:31:45.468651 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.468626    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"]
Sep 15 21:31:45.589455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.589418    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tls-certs\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.589613 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.589464    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-home\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.589613 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.589562    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-model-cache\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.589613 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.589603    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tmp-dir\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.589726 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.589662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-dshm\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.589761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.589739    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsc8\" (UniqueName: \"kubernetes.io/projected/0a15df89-7f7f-4596-bc62-cfd6346a8caf-kube-api-access-jrsc8\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.690878 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.690823    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsc8\" (UniqueName: \"kubernetes.io/projected/0a15df89-7f7f-4596-bc62-cfd6346a8caf-kube-api-access-jrsc8\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.691095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.690897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tls-certs\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.691095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.690945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-home\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.691095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.691023    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-model-cache\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.691095 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.691062    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tmp-dir\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.691348 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.691120    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-dshm\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.693484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.693456    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-home\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.693648 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.693522    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-model-cache\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.693768 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.693741    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tmp-dir\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.695435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.695414    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-dshm\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.696223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.696176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tls-certs\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.702912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.702886    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsc8\" (UniqueName: \"kubernetes.io/projected/0a15df89-7f7f-4596-bc62-cfd6346a8caf-kube-api-access-jrsc8\") pod \"e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") " pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:45.770283 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:45.770205    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:46.050024 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:46.049987    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"]
Sep 15 21:31:46.051257 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:31:46.051166    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a15df89_7f7f_4596_bc62_cfd6346a8caf.slice/crio-a3cdf2470abf7ce974c70660b67a242b34d1cd5a145aac9553a8cbba7322ca48 WatchSource:0}: Error finding container a3cdf2470abf7ce974c70660b67a242b34d1cd5a145aac9553a8cbba7322ca48: Status 404 returned error can't find the container with id a3cdf2470abf7ce974c70660b67a242b34d1cd5a145aac9553a8cbba7322ca48
Sep 15 21:31:46.318116 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:46.318023    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw" event={"ID":"0a15df89-7f7f-4596-bc62-cfd6346a8caf","Type":"ContainerStarted","Data":"723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3"}
Sep 15 21:31:46.318116 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:46.318072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw" event={"ID":"0a15df89-7f7f-4596-bc62-cfd6346a8caf","Type":"ContainerStarted","Data":"a3cdf2470abf7ce974c70660b67a242b34d1cd5a145aac9553a8cbba7322ca48"}
Sep 15 21:31:46.350354 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:46.350300    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw" podStartSLOduration=1.350281212 podStartE2EDuration="1.350281212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:46.348915453 +0000 UTC m=+1405.150784395" watchObservedRunningTime="2026-09-15 21:31:46.350281212 +0000 UTC m=+1405.152150155"
Sep 15 21:31:49.774046 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:49.774009    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"]
Sep 15 21:31:49.774550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:49.774263    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy" containerID="cri-o://195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e" gracePeriod=30
Sep 15 21:31:50.646905 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.646864    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"]
Sep 15 21:31:50.649723 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.649701    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.652462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.652439    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3c4dc8-openshift-default-dockercfg-drzwj\""
Sep 15 21:31:50.662180 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.662153    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"]
Sep 15 21:31:50.746625 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.746556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-certs\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.746994 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.746969    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-podinfo\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.777013 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.776979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28668168-ee46-4e5b-8401-d61c6ba9076f-istiod-ca-cert\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.777619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.777594    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-data\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.777735 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.777719    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxq2\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-kube-api-access-bfxq2\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.777973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.777955    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-envoy\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.778111 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.778097    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-socket\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.778455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.778435    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-token\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.778592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.778572    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-credential-socket\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880064 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880023    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-token\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880077    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-credential-socket\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880205    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-certs\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-podinfo\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880265    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28668168-ee46-4e5b-8401-d61c6ba9076f-istiod-ca-cert\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-data\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880632 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxq2\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-kube-api-access-bfxq2\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880632 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880465    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-envoy\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880632 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-socket\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880632 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880574    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-credential-socket\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880632 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880622    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-certs\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.880900 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880817    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-data\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.881020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.880906    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-socket\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.881135 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.881114    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28668168-ee46-4e5b-8401-d61c6ba9076f-istiod-ca-cert\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.882789 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.882766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-envoy\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.883049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.883033    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-podinfo\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.891526 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.891502    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-token\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.892012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.891992    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxq2\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-kube-api-access-bfxq2\") pod \"e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") " pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:50.962439 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:50.962405    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:51.245113 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.245086    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"]
Sep 15 21:31:51.246860 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:31:51.246833    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28668168_ee46_4e5b_8401_d61c6ba9076f.slice/crio-2a954264a1f6641fd1d4527e17340dc9b3fc6e449b5effd2feb14affc6f9e1c1 WatchSource:0}: Error finding container 2a954264a1f6641fd1d4527e17340dc9b3fc6e449b5effd2feb14affc6f9e1c1: Status 404 returned error can't find the container with id 2a954264a1f6641fd1d4527e17340dc9b3fc6e449b5effd2feb14affc6f9e1c1
Sep 15 21:31:51.249341 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.249305    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:31:51.249438 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.249372    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:31:51.249438 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.249399    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:31:51.349809 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.349771    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" event={"ID":"28668168-ee46-4e5b-8401-d61c6ba9076f","Type":"ContainerStarted","Data":"1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"}
Sep 15 21:31:51.350012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.349830    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" event={"ID":"28668168-ee46-4e5b-8401-d61c6ba9076f","Type":"ContainerStarted","Data":"2a954264a1f6641fd1d4527e17340dc9b3fc6e449b5effd2feb14affc6f9e1c1"}
Sep 15 21:31:51.369849 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.369788    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podStartSLOduration=1.369767915 podStartE2EDuration="1.369767915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:51.36897771 +0000 UTC m=+1410.170846680" watchObservedRunningTime="2026-09-15 21:31:51.369767915 +0000 UTC m=+1410.171636863"
Sep 15 21:31:51.962870 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:51.962839    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:52.178782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:52.178746    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"]
Sep 15 21:31:52.968251 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:52.968221    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:53.360663 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:53.360573    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:53.361653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:53.361636    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:31:55.033729 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.033697    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:31:55.229565 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229532    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c283b47-2030-4956-b896-d6773ec132c6-istio-podinfo\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.229742 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229587    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-certs\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.229742 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229622    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-credential-socket\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.229742 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229727    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-envoy\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.229907 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229786    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbmg\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-kube-api-access-lkbmg\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.229907 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229826    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-socket\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.230025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229901    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:55.230025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229913    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:55.230025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229908    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-istio-token\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.230025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.229998    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c283b47-2030-4956-b896-d6773ec132c6-istiod-ca-cert\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.230408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230046    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-data\") pod \"5c283b47-2030-4956-b896-d6773ec132c6\" (UID: \"5c283b47-2030-4956-b896-d6773ec132c6\") "
Sep 15 21:31:55.230408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230129    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:55.230408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230387    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:55.230748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230559    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.230748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230573    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.230748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230585    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.230748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230593    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.230748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.230566    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c283b47-2030-4956-b896-d6773ec132c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:55.231925 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.231904    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c283b47-2030-4956-b896-d6773ec132c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:31:55.232048 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.232032    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:31:55.232554 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.232531    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-kube-api-access-lkbmg" (OuterVolumeSpecName: "kube-api-access-lkbmg") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "kube-api-access-lkbmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:55.232622 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.232548    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c283b47-2030-4956-b896-d6773ec132c6" (UID: "5c283b47-2030-4956-b896-d6773ec132c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:55.331669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.331635    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.331669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.331663    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c283b47-2030-4956-b896-d6773ec132c6-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.331669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.331671    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c283b47-2030-4956-b896-d6773ec132c6-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.331669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.331678    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c283b47-2030-4956-b896-d6773ec132c6-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.331928 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.331687    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbmg\" (UniqueName: \"kubernetes.io/projected/5c283b47-2030-4956-b896-d6773ec132c6-kube-api-access-lkbmg\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:31:55.374413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.374377    2551 generic.go:356] "Generic (PLEG): container finished" podID="5c283b47-2030-4956-b896-d6773ec132c6" containerID="195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e" exitCode=0
Sep 15 21:31:55.374588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.374430    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"
Sep 15 21:31:55.374588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.374454    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" event={"ID":"5c283b47-2030-4956-b896-d6773ec132c6","Type":"ContainerDied","Data":"195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e"}
Sep 15 21:31:55.374588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.374535    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr" event={"ID":"5c283b47-2030-4956-b896-d6773ec132c6","Type":"ContainerDied","Data":"645e7a689e1eaf404913bebb2f23b8d2811bcec182ede0da65030291fa0f090d"}
Sep 15 21:31:55.374588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.374557    2551 scope.go:117] "RemoveContainer" containerID="195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e"
Sep 15 21:31:55.384619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.384600    2551 scope.go:117] "RemoveContainer" containerID="21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"
Sep 15 21:31:55.392908 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.392891    2551 scope.go:117] "RemoveContainer" containerID="195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e"
Sep 15 21:31:55.393184 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:31:55.393167    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e\": container with ID starting with 195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e not found: ID does not exist" containerID="195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e"
Sep 15 21:31:55.393248 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.393211    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e"} err="failed to get container status \"195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e\": rpc error: code = NotFound desc = could not find container \"195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e\": container with ID starting with 195ff0812901b86b3a0761dd17afcaab37fbb594ca7fd78914d7b061dc46796e not found: ID does not exist"
Sep 15 21:31:55.393248 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.393232    2551 scope.go:117] "RemoveContainer" containerID="21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"
Sep 15 21:31:55.393480 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:31:55.393466    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e\": container with ID starting with 21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e not found: ID does not exist" containerID="21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"
Sep 15 21:31:55.393518 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.393484    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e"} err="failed to get container status \"21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e\": rpc error: code = NotFound desc = could not find container \"21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e\": container with ID starting with 21c17db32ba7ec9c32eec07ec3d5b998aa994e350e3e6e1bb70beda754a9ab3e not found: ID does not exist"
Sep 15 21:31:55.399425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.399396    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"]
Sep 15 21:31:55.405286 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.405267    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-779ace-openshift-default-c7dd699d4-vpmfr"]
Sep 15 21:31:55.769163 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.769131    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c283b47-2030-4956-b896-d6773ec132c6" path="/var/lib/kubelet/pods/5c283b47-2030-4956-b896-d6773ec132c6/volumes"
Sep 15 21:31:55.771136 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.771116    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:55.771260 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.771161    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:55.784154 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:55.784123    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:31:56.392274 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:31:56.392243    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:32:07.198768 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.198704    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm" podUID="38adf8c0-69c3-4bab-8aff-12aae31f8eae" containerName="main" containerID="cri-o://0616d5fd851f7d0e2b227e8e28ff201af95214a7d01f2152d444889c0fc43eae" gracePeriod=45
Sep 15 21:32:07.439114 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.439084    2551 generic.go:356] "Generic (PLEG): container finished" podID="38adf8c0-69c3-4bab-8aff-12aae31f8eae" containerID="0616d5fd851f7d0e2b227e8e28ff201af95214a7d01f2152d444889c0fc43eae" exitCode=0
Sep 15 21:32:07.439265 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.439127    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm" event={"ID":"38adf8c0-69c3-4bab-8aff-12aae31f8eae","Type":"ContainerDied","Data":"0616d5fd851f7d0e2b227e8e28ff201af95214a7d01f2152d444889c0fc43eae"}
Sep 15 21:32:07.455380 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.455320    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:32:07.553428 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553384    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-home\") pod \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") "
Sep 15 21:32:07.553428 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553440    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tmp-dir\") pod \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") "
Sep 15 21:32:07.553638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553549    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-dshm\") pod \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") "
Sep 15 21:32:07.553638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553598    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9jnd\" (UniqueName: \"kubernetes.io/projected/38adf8c0-69c3-4bab-8aff-12aae31f8eae-kube-api-access-g9jnd\") pod \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") "
Sep 15 21:32:07.553638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553626    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-model-cache\") pod \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") "
Sep 15 21:32:07.553782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553669    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-home" (OuterVolumeSpecName: "home") pod "38adf8c0-69c3-4bab-8aff-12aae31f8eae" (UID: "38adf8c0-69c3-4bab-8aff-12aae31f8eae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:07.553782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553699    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tls-certs\") pod \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\" (UID: \"38adf8c0-69c3-4bab-8aff-12aae31f8eae\") "
Sep 15 21:32:07.553782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553750    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38adf8c0-69c3-4bab-8aff-12aae31f8eae" (UID: "38adf8c0-69c3-4bab-8aff-12aae31f8eae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:07.553985 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.553959    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-model-cache" (OuterVolumeSpecName: "model-cache") pod "38adf8c0-69c3-4bab-8aff-12aae31f8eae" (UID: "38adf8c0-69c3-4bab-8aff-12aae31f8eae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:07.554390 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.554275    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:32:07.554462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.554395    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:32:07.554462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.554433    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:32:07.555859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.555838    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38adf8c0-69c3-4bab-8aff-12aae31f8eae" (UID: "38adf8c0-69c3-4bab-8aff-12aae31f8eae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:32:07.555927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.555905    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38adf8c0-69c3-4bab-8aff-12aae31f8eae-kube-api-access-g9jnd" (OuterVolumeSpecName: "kube-api-access-g9jnd") pod "38adf8c0-69c3-4bab-8aff-12aae31f8eae" (UID: "38adf8c0-69c3-4bab-8aff-12aae31f8eae"). InnerVolumeSpecName "kube-api-access-g9jnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:07.555978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.555918    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-dshm" (OuterVolumeSpecName: "dshm") pod "38adf8c0-69c3-4bab-8aff-12aae31f8eae" (UID: "38adf8c0-69c3-4bab-8aff-12aae31f8eae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:07.655592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.655556    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38adf8c0-69c3-4bab-8aff-12aae31f8eae-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:32:07.655592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.655587    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jnd\" (UniqueName: \"kubernetes.io/projected/38adf8c0-69c3-4bab-8aff-12aae31f8eae-kube-api-access-g9jnd\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:32:07.655796 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:07.655603    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38adf8c0-69c3-4bab-8aff-12aae31f8eae-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:32:08.446634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:08.446602    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"
Sep 15 21:32:08.447115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:08.446639    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm" event={"ID":"38adf8c0-69c3-4bab-8aff-12aae31f8eae","Type":"ContainerDied","Data":"ac5c09b2e408576a34f9578e61e905fb60e460c27555452e7724e5cb501e962d"}
Sep 15 21:32:08.447115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:08.446683    2551 scope.go:117] "RemoveContainer" containerID="0616d5fd851f7d0e2b227e8e28ff201af95214a7d01f2152d444889c0fc43eae"
Sep 15 21:32:08.464470 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:08.464441    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"]
Sep 15 21:32:08.469113 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:08.469090    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d130a8/noten-model-d130a8-kserve-67c9b76b87-v22wm"]
Sep 15 21:32:09.769808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:09.769777    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38adf8c0-69c3-4bab-8aff-12aae31f8eae" path="/var/lib/kubelet/pods/38adf8c0-69c3-4bab-8aff-12aae31f8eae/volumes"
Sep 15 21:32:20.523519 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.523483    2551 generic.go:356] "Generic (PLEG): container finished" podID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerID="1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623" exitCode=0
Sep 15 21:32:20.524041 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.523563    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" event={"ID":"28668168-ee46-4e5b-8401-d61c6ba9076f","Type":"ContainerDied","Data":"1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"}
Sep 15 21:32:20.524124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.524106    2551 scope.go:117] "RemoveContainer" containerID="1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"
Sep 15 21:32:20.525338 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.525312    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:32:20.525422 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.525408    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:32:20.525465 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.525452    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:32:20.729270 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729235    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"]
Sep 15 21:32:20.729664 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729649    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy"
Sep 15 21:32:20.729664 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729665    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729682    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38adf8c0-69c3-4bab-8aff-12aae31f8eae" containerName="main"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729687    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="38adf8c0-69c3-4bab-8aff-12aae31f8eae" containerName="main"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729694    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729699    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729763    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="38adf8c0-69c3-4bab-8aff-12aae31f8eae" containerName="main"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729770    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy"
Sep 15 21:32:20.729817 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.729778    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c283b47-2030-4956-b896-d6773ec132c6" containerName="istio-proxy"
Sep 15 21:32:20.732028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.732007    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.734954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.734926    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"default-dockercfg-ksdmf\""
Sep 15 21:32:20.735060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.735034    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"openshift-service-ca.crt\""
Sep 15 21:32:20.735119 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.734930    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"kube-root-ca.crt\""
Sep 15 21:32:20.735288 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.735179    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"distinct-2-w0-a276f9-kserve-self-signed-certs\""
Sep 15 21:32:20.745812 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.745787    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"]
Sep 15 21:32:20.795212 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.795106    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-model-cache\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.795212 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.795146    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-home\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.795212 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.795209    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-dshm\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.795462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.795259    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-tmp-dir\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.795462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.795354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882a2d76-cecd-49d9-b092-b777907ce81f-tls-certs\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.795462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.795415    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2df\" (UniqueName: \"kubernetes.io/projected/882a2d76-cecd-49d9-b092-b777907ce81f-kube-api-access-vq2df\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.896861 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.896822    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882a2d76-cecd-49d9-b092-b777907ce81f-tls-certs\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.896861 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.896867    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2df\" (UniqueName: \"kubernetes.io/projected/882a2d76-cecd-49d9-b092-b777907ce81f-kube-api-access-vq2df\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.896925    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-model-cache\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.896942    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-home\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.896975    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-dshm\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.896995    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-tmp-dir\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897372 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.897342    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-model-cache\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.897405    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-home\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.897491 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.897449    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-tmp-dir\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.899281 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.899265    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-dshm\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.899544 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.899526    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882a2d76-cecd-49d9-b092-b777907ce81f-tls-certs\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:20.909368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:20.909332    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2df\" (UniqueName: \"kubernetes.io/projected/882a2d76-cecd-49d9-b092-b777907ce81f-kube-api-access-vq2df\") pod \"distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") " pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:21.046076 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.045982    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:21.342517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.342492    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"]
Sep 15 21:32:21.344254 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:32:21.344226    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod882a2d76_cecd_49d9_b092_b777907ce81f.slice/crio-12b1112a66557f137901130d2c29a1587f46d6a8c32a2388aa332f539161984b WatchSource:0}: Error finding container 12b1112a66557f137901130d2c29a1587f46d6a8c32a2388aa332f539161984b: Status 404 returned error can't find the container with id 12b1112a66557f137901130d2c29a1587f46d6a8c32a2388aa332f539161984b
Sep 15 21:32:21.530555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.530513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm" event={"ID":"882a2d76-cecd-49d9-b092-b777907ce81f","Type":"ContainerStarted","Data":"d8b7ad4341f6e92ee1b47af37c0a1840a7586855b79dc8c7fafe4f7010d8a35f"}
Sep 15 21:32:21.531000 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.530566    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm" event={"ID":"882a2d76-cecd-49d9-b092-b777907ce81f","Type":"ContainerStarted","Data":"12b1112a66557f137901130d2c29a1587f46d6a8c32a2388aa332f539161984b"}
Sep 15 21:32:21.532678 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.532652    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" event={"ID":"28668168-ee46-4e5b-8401-d61c6ba9076f","Type":"ContainerStarted","Data":"f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0"}
Sep 15 21:32:21.547113 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.547065    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm" podStartSLOduration=1.547046 podStartE2EDuration="1.547046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:21.546316375 +0000 UTC m=+1440.348185331" watchObservedRunningTime="2026-09-15 21:32:21.547046 +0000 UTC m=+1440.348914947"
Sep 15 21:32:21.962644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.962605    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:32:21.963900 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.963874    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:32:21.964002 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:21.963938    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:32:22.963685 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:22.963649    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:32:22.964171 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:22.963716    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:32:23.361512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:23.361413    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:32:23.963801 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:23.963768    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:32:23.964225 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:23.963831    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:32:24.002181 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.002138    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"]
Sep 15 21:32:24.006757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.006730    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.009349 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.009322    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7c57b6-openshift-default-dockercfg-mdst8\""
Sep 15 21:32:24.016072 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.015950    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"]
Sep 15 21:32:24.131507 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131461    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-envoy\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.131507 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131508    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-data\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.131780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131526    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-credential-socket\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.131780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131565    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-socket\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.131780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xzb\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-kube-api-access-s4xzb\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.131780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131676    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-certs\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.131780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-token\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.132011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131803    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52c3134-abb9-4192-bc05-23fc31e162b8-istiod-ca-cert\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.132011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.131846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-podinfo\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.233726 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.233617    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52c3134-abb9-4192-bc05-23fc31e162b8-istiod-ca-cert\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.235588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.235042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-podinfo\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.235935 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.234935    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52c3134-abb9-4192-bc05-23fc31e162b8-istiod-ca-cert\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.236099 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.236079    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-envoy\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.236274 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.236257    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-data\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.236403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.236387    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-credential-socket\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.236580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.236566    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-socket\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.236738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.236719    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xzb\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-kube-api-access-s4xzb\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.237051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.237033    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-certs\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.237226 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.237209    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-token\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.237429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.237371    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-certs\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.237429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.236872    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-socket\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.237429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.237112    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-credential-socket\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.237641 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.237621    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-data\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.240750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.240720    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-podinfo\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.240872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.240784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-envoy\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.246383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.246350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xzb\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-kube-api-access-s4xzb\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.246820 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.246778    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-token\") pod \"e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") " pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.319367 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.319331    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:24.623349 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.623320    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"]
Sep 15 21:32:24.625203 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:32:24.625159    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52c3134_abb9_4192_bc05_23fc31e162b8.slice/crio-50215bcf927dfc6732a1e83f54e285fd3a55e4cf3887fc3684e238719011a0f5 WatchSource:0}: Error finding container 50215bcf927dfc6732a1e83f54e285fd3a55e4cf3887fc3684e238719011a0f5: Status 404 returned error can't find the container with id 50215bcf927dfc6732a1e83f54e285fd3a55e4cf3887fc3684e238719011a0f5
Sep 15 21:32:24.627691 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.627658    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:32:24.627764 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.627729    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:32:24.627802 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.627762    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:32:24.962768 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.962723    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:32:24.962986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:24.962787    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:32:25.558163 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:25.558124    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" event={"ID":"d52c3134-abb9-4192-bc05-23fc31e162b8","Type":"ContainerStarted","Data":"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"}
Sep 15 21:32:25.558656 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:25.558234    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" event={"ID":"d52c3134-abb9-4192-bc05-23fc31e162b8","Type":"ContainerStarted","Data":"50215bcf927dfc6732a1e83f54e285fd3a55e4cf3887fc3684e238719011a0f5"}
Sep 15 21:32:25.581463 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:25.581381    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podStartSLOduration=2.581345052 podStartE2EDuration="2.581345052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:25.57872635 +0000 UTC m=+1444.380595290" watchObservedRunningTime="2026-09-15 21:32:25.581345052 +0000 UTC m=+1444.383213996"
Sep 15 21:32:25.963540 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:25.963505    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 21:32:25.963741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:25.963569    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 21:32:26.320455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:26.320358    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:26.325722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:26.325696    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:26.564091 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:26.564055    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:26.565070 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:26.565050    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:32:26.968162 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:26.968119    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:32:26.969100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:26.969079    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:32:31.046814 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:31.046780    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:31.047269 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:31.046828    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:31.059147 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:31.059124    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:32:31.600388 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:32:31.600355    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:33:02.764739 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:02.764707    2551 generic.go:356] "Generic (PLEG): container finished" podID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerID="9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f" exitCode=0
Sep 15 21:33:02.765309 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:02.764782    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" event={"ID":"d52c3134-abb9-4192-bc05-23fc31e162b8","Type":"ContainerDied","Data":"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"}
Sep 15 21:33:02.765386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:02.765371    2551 scope.go:117] "RemoveContainer" containerID="9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"
Sep 15 21:33:02.766591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:02.766560    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:33:02.766682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:02.766633    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:33:02.766682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:02.766666    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:33:03.775498 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:03.775452    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" event={"ID":"d52c3134-abb9-4192-bc05-23fc31e162b8","Type":"ContainerStarted","Data":"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"}
Sep 15 21:33:04.319784 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:04.319753    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:33:04.319955 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:04.319927    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:33:04.320005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:04.319972    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 15 21:33:05.319699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:05.319657    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:33:05.320185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:05.319712    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 15 21:33:06.320057 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:06.320018    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:33:06.320460 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:06.320093    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 15 21:33:07.320419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:07.320382    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:33:07.320918 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:07.320479    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 15 21:33:08.320402 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:08.320365    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 15 21:33:08.320580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:08.320427    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 15 21:33:09.324262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:09.324231    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:33:09.324742 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:09.324606    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:33:09.325423 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:09.325401    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:33:38.925898 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.925858    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"]
Sep 15 21:33:38.929258 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.929235    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.932495 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.932471    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c4dc8\"/\"default-dockercfg-8r4z7\""
Sep 15 21:33:38.932495 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.932487    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c4dc8\"/\"kube-root-ca.crt\""
Sep 15 21:33:38.932672 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.932476    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c4dc8\"/\"auth-test-model-3c4dc8-kserve-self-signed-certs\""
Sep 15 21:33:38.932672 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.932478    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c4dc8\"/\"openshift-service-ca.crt\""
Sep 15 21:33:38.938032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.937852    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-dshm\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.938032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.937937    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmh4\" (UniqueName: \"kubernetes.io/projected/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-kube-api-access-jbmh4\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.938032 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.938015    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tls-certs\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.938241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.938067    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-home\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.938241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.938124    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tmp-dir\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.938241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.938185    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-model-cache\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:38.947545 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:38.947515    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"]
Sep 15 21:33:39.039256 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-dshm\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039283    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmh4\" (UniqueName: \"kubernetes.io/projected/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-kube-api-access-jbmh4\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039312    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tls-certs\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039335    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-home\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039365    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tmp-dir\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039469 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039400    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-model-cache\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039791 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-home\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039805    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tmp-dir\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.039923 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.039868    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-model-cache\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.041697 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.041679    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-dshm\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.041994 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.041978    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tls-certs\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.047889 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.047866    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmh4\" (UniqueName: \"kubernetes.io/projected/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-kube-api-access-jbmh4\") pod \"auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") " pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.248241 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.248125    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:39.535104 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.535070    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"]
Sep 15 21:33:39.536259 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:33:39.536231    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode49dd3a5_933f_42ab_a863_c45c3af0a7f9.slice/crio-78756dcda4723a44d75ff1d9a98690e57a3e3a1366c0cb332dee9e4744eaff0a WatchSource:0}: Error finding container 78756dcda4723a44d75ff1d9a98690e57a3e3a1366c0cb332dee9e4744eaff0a: Status 404 returned error can't find the container with id 78756dcda4723a44d75ff1d9a98690e57a3e3a1366c0cb332dee9e4744eaff0a
Sep 15 21:33:39.978085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.978022    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l" event={"ID":"e49dd3a5-933f-42ab-a863-c45c3af0a7f9","Type":"ContainerStarted","Data":"5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2"}
Sep 15 21:33:39.978085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.978090    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l" event={"ID":"e49dd3a5-933f-42ab-a863-c45c3af0a7f9","Type":"ContainerStarted","Data":"78756dcda4723a44d75ff1d9a98690e57a3e3a1366c0cb332dee9e4744eaff0a"}
Sep 15 21:33:39.996400 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:39.996261    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l" podStartSLOduration=1.996239168 podStartE2EDuration="1.996239168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:39.993970899 +0000 UTC m=+1518.795839843" watchObservedRunningTime="2026-09-15 21:33:39.996239168 +0000 UTC m=+1518.798108112"
Sep 15 21:33:49.248351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:49.248264    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:49.248777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:49.248397    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:49.261299 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:49.261277    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:33:50.039883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:33:50.039846    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:34:02.723020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:02.722986    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"]
Sep 15 21:34:17.743216 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:17.743137    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw" podUID="0a15df89-7f7f-4596-bc62-cfd6346a8caf" containerName="main" containerID="cri-o://723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3" gracePeriod=45
Sep 15 21:34:17.993703 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:17.993641    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:34:18.068225 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068153    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrsc8\" (UniqueName: \"kubernetes.io/projected/0a15df89-7f7f-4596-bc62-cfd6346a8caf-kube-api-access-jrsc8\") pod \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") "
Sep 15 21:34:18.068422 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068251    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-model-cache\") pod \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") "
Sep 15 21:34:18.068422 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-home\") pod \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") "
Sep 15 21:34:18.068422 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068359    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-dshm\") pod \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") "
Sep 15 21:34:18.068598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068436    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tmp-dir\") pod \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") "
Sep 15 21:34:18.068598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068479    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tls-certs\") pod \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\" (UID: \"0a15df89-7f7f-4596-bc62-cfd6346a8caf\") "
Sep 15 21:34:18.068598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068576    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a15df89-7f7f-4596-bc62-cfd6346a8caf" (UID: "0a15df89-7f7f-4596-bc62-cfd6346a8caf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:18.068736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068608    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-home" (OuterVolumeSpecName: "home") pod "0a15df89-7f7f-4596-bc62-cfd6346a8caf" (UID: "0a15df89-7f7f-4596-bc62-cfd6346a8caf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:18.068884 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068858    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:18.068884 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068881    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:18.069067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.068933    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a15df89-7f7f-4596-bc62-cfd6346a8caf" (UID: "0a15df89-7f7f-4596-bc62-cfd6346a8caf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:18.070915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.070886    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a15df89-7f7f-4596-bc62-cfd6346a8caf" (UID: "0a15df89-7f7f-4596-bc62-cfd6346a8caf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:34:18.071234 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.071179    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a15df89-7f7f-4596-bc62-cfd6346a8caf-kube-api-access-jrsc8" (OuterVolumeSpecName: "kube-api-access-jrsc8") pod "0a15df89-7f7f-4596-bc62-cfd6346a8caf" (UID: "0a15df89-7f7f-4596-bc62-cfd6346a8caf"). InnerVolumeSpecName "kube-api-access-jrsc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:18.071234 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.071212    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-dshm" (OuterVolumeSpecName: "dshm") pod "0a15df89-7f7f-4596-bc62-cfd6346a8caf" (UID: "0a15df89-7f7f-4596-bc62-cfd6346a8caf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:18.169596 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.169560    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:18.169596 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.169590    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a15df89-7f7f-4596-bc62-cfd6346a8caf-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:18.169596 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.169599    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrsc8\" (UniqueName: \"kubernetes.io/projected/0a15df89-7f7f-4596-bc62-cfd6346a8caf-kube-api-access-jrsc8\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:18.169835 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.169608    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a15df89-7f7f-4596-bc62-cfd6346a8caf-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:18.188676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.188643    2551 generic.go:356] "Generic (PLEG): container finished" podID="0a15df89-7f7f-4596-bc62-cfd6346a8caf" containerID="723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3" exitCode=0
Sep 15 21:34:18.188847 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.188691    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"
Sep 15 21:34:18.188847 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.188738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw" event={"ID":"0a15df89-7f7f-4596-bc62-cfd6346a8caf","Type":"ContainerDied","Data":"723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3"}
Sep 15 21:34:18.188847 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.188783    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw" event={"ID":"0a15df89-7f7f-4596-bc62-cfd6346a8caf","Type":"ContainerDied","Data":"a3cdf2470abf7ce974c70660b67a242b34d1cd5a145aac9553a8cbba7322ca48"}
Sep 15 21:34:18.188847 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.188802    2551 scope.go:117] "RemoveContainer" containerID="723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3"
Sep 15 21:34:18.198352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.198329    2551 scope.go:117] "RemoveContainer" containerID="723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3"
Sep 15 21:34:18.198643 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:34:18.198624    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3\": container with ID starting with 723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3 not found: ID does not exist" containerID="723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3"
Sep 15 21:34:18.198693 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.198652    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3"} err="failed to get container status \"723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3\": rpc error: code = NotFound desc = could not find container \"723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3\": container with ID starting with 723360032d209b91c0a10fd1f038f239152737bb53e2af2f3d45993f4d9b59c3 not found: ID does not exist"
Sep 15 21:34:18.211285 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.211256    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"]
Sep 15 21:34:18.214094 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:18.214074    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-625609/e2e-conflict-b-23fd2eb6-kserve-68b7d9c587-wrnvw"]
Sep 15 21:34:19.532338 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.532304    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"]
Sep 15 21:34:19.532848 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.532551    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k" podUID="543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" containerName="maas-api" containerID="cri-o://6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8" gracePeriod=30
Sep 15 21:34:19.772250 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.772215    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a15df89-7f7f-4596-bc62-cfd6346a8caf" path="/var/lib/kubelet/pods/0a15df89-7f7f-4596-bc62-cfd6346a8caf/volumes"
Sep 15 21:34:19.792974 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.792916    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:34:19.887294 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.887261    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-metrics-certs\") pod \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") "
Sep 15 21:34:19.887483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.887338    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-maas-api-tls\") pod \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") "
Sep 15 21:34:19.887483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.887451    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tbsl\" (UniqueName: \"kubernetes.io/projected/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-kube-api-access-2tbsl\") pod \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") "
Sep 15 21:34:19.887621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.887484    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-openshift-service-ca\") pod \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\" (UID: \"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286\") "
Sep 15 21:34:19.887944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.887865    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" (UID: "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:34:19.889517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.889491    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" (UID: "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:34:19.889643 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.889631    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-kube-api-access-2tbsl" (OuterVolumeSpecName: "kube-api-access-2tbsl") pod "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" (UID: "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286"). InnerVolumeSpecName "kube-api-access-2tbsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:19.889913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.889897    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" (UID: "543a10d3-ca8e-4f69-bbfe-aaa9bc90c286"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:34:19.988507 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.988473    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-metrics-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:19.988507 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.988502    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-maas-api-tls\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:19.988507 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.988511    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tbsl\" (UniqueName: \"kubernetes.io/projected/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-kube-api-access-2tbsl\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:19.988807 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:19.988521    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286-openshift-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:34:20.201532 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.201497    2551 generic.go:356] "Generic (PLEG): container finished" podID="543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" containerID="6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8" exitCode=0
Sep 15 21:34:20.201710 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.201544    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"
Sep 15 21:34:20.201710 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.201581    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k" event={"ID":"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286","Type":"ContainerDied","Data":"6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8"}
Sep 15 21:34:20.201710 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.201634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k" event={"ID":"543a10d3-ca8e-4f69-bbfe-aaa9bc90c286","Type":"ContainerDied","Data":"709b61ca15b1e42f3efd450e67a1de2cb3be6529491785b0c20a0dfb97590353"}
Sep 15 21:34:20.201710 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.201656    2551 scope.go:117] "RemoveContainer" containerID="6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8"
Sep 15 21:34:20.211772 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.211757    2551 scope.go:117] "RemoveContainer" containerID="6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8"
Sep 15 21:34:20.212062 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:34:20.212046    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8\": container with ID starting with 6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8 not found: ID does not exist" containerID="6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8"
Sep 15 21:34:20.212119 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.212071    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8"} err="failed to get container status \"6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8\": rpc error: code = NotFound desc = could not find container \"6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8\": container with ID starting with 6b47632aa683cb25760e11e9ff8c96a871eba5e2f5f1b1e1107b3578d05145c8 not found: ID does not exist"
Sep 15 21:34:20.224837 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.224815    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"]
Sep 15 21:34:20.228921 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:20.228895    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e476d59-5c5c694dcc-jhk6k"]
Sep 15 21:34:21.769464 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:21.769434    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" path="/var/lib/kubelet/pods/543a10d3-ca8e-4f69-bbfe-aaa9bc90c286/volumes"
Sep 15 21:34:26.345845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.345804    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"]
Sep 15 21:34:26.346432 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.346413    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" containerName="maas-api"
Sep 15 21:34:26.346523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.346433    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" containerName="maas-api"
Sep 15 21:34:26.346523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.346449    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a15df89-7f7f-4596-bc62-cfd6346a8caf" containerName="main"
Sep 15 21:34:26.346523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.346456    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a15df89-7f7f-4596-bc62-cfd6346a8caf" containerName="main"
Sep 15 21:34:26.346667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.346547    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a15df89-7f7f-4596-bc62-cfd6346a8caf" containerName="main"
Sep 15 21:34:26.346667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.346565    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="543a10d3-ca8e-4f69-bbfe-aaa9bc90c286" containerName="maas-api"
Sep 15 21:34:26.350931 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.350909    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.353692 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.353672    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-625609\"/\"distinct-w4-625609-kserve-self-signed-certs\""
Sep 15 21:34:26.359877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.359854    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"]
Sep 15 21:34:26.461783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.461742    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tmp-dir\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.461975 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.461789    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tls-certs\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.461975 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.461862    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rlw\" (UniqueName: \"kubernetes.io/projected/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-kube-api-access-x7rlw\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.461975 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.461895    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-dshm\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.462148 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.462047    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-home\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.462289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.462142    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-model-cache\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563266 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563230    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-home\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-model-cache\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563326    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tmp-dir\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tls-certs\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563472 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rlw\" (UniqueName: \"kubernetes.io/projected/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-kube-api-access-x7rlw\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563686 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563602    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-dshm\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563686 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-home\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563791 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563708    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tmp-dir\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.563791 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.563764    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-model-cache\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.565898 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.565869    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-dshm\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.566216 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.566170    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tls-certs\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.572561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.572537    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rlw\" (UniqueName: \"kubernetes.io/projected/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-kube-api-access-x7rlw\") pod \"distinct-w4-625609-kserve-6fd75874c9-gg6gt\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") " pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.661111 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.661085    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:26.956526 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:26.956498    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"]
Sep 15 21:34:26.958180 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:34:26.958151    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec29a1f6_8d6a_4112_a1ff_d4935173ad0f.slice/crio-98579f27c862af772e8a5d6f63d38794cc728191eec7c58acbe27ad90fe63eb1 WatchSource:0}: Error finding container 98579f27c862af772e8a5d6f63d38794cc728191eec7c58acbe27ad90fe63eb1: Status 404 returned error can't find the container with id 98579f27c862af772e8a5d6f63d38794cc728191eec7c58acbe27ad90fe63eb1
Sep 15 21:34:27.241926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:27.241820    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt" event={"ID":"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f","Type":"ContainerStarted","Data":"1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115"}
Sep 15 21:34:27.241926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:27.241866    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt" event={"ID":"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f","Type":"ContainerStarted","Data":"98579f27c862af772e8a5d6f63d38794cc728191eec7c58acbe27ad90fe63eb1"}
Sep 15 21:34:27.260337 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:27.260285    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt" podStartSLOduration=1.260267102 podStartE2EDuration="1.260267102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:27.259488259 +0000 UTC m=+1566.061357225" watchObservedRunningTime="2026-09-15 21:34:27.260267102 +0000 UTC m=+1566.062136045"
Sep 15 21:34:36.662120 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:36.662082    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:36.662580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:36.662138    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:36.676681 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:36.676652    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:34:37.306132 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:34:37.306103    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:35:17.165169 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:17.165086    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"]
Sep 15 21:35:17.166121 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:17.165320    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy" containerID="cri-o://5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc" gracePeriod=30
Sep 15 21:35:22.414114 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.414083    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:35:22.543050 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.542947    2551 generic.go:356] "Generic (PLEG): container finished" podID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerID="5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc" exitCode=0
Sep 15 21:35:22.543050 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.542993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" event={"ID":"9b90fcf5-05dc-4164-b3fc-4342e91055d3","Type":"ContainerDied","Data":"5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc"}
Sep 15 21:35:22.543050 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.543043    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26" event={"ID":"9b90fcf5-05dc-4164-b3fc-4342e91055d3","Type":"ContainerDied","Data":"b2ef9179ad0872ff9facd53b6ac5ea60cfeba269468eaff67946aad0630fdb5e"}
Sep 15 21:35:22.543358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.543073    2551 scope.go:117] "RemoveContainer" containerID="5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc"
Sep 15 21:35:22.543358 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.542997    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"
Sep 15 21:35:22.554289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.554267    2551 scope.go:117] "RemoveContainer" containerID="32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"
Sep 15 21:35:22.562901 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.562876    2551 scope.go:117] "RemoveContainer" containerID="5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc"
Sep 15 21:35:22.563174 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:35:22.563154    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc\": container with ID starting with 5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc not found: ID does not exist" containerID="5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc"
Sep 15 21:35:22.563265 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.563206    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc"} err="failed to get container status \"5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc\": rpc error: code = NotFound desc = could not find container \"5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc\": container with ID starting with 5d756047cedfc4c4420fbc6fec97b724d7cc4e8b6b9a0c472813b4ca3a134fbc not found: ID does not exist"
Sep 15 21:35:22.563265 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.563226    2551 scope.go:117] "RemoveContainer" containerID="32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"
Sep 15 21:35:22.563512 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:35:22.563494    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c\": container with ID starting with 32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c not found: ID does not exist" containerID="32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"
Sep 15 21:35:22.563565 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.563517    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c"} err="failed to get container status \"32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c\": rpc error: code = NotFound desc = could not find container \"32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c\": container with ID starting with 32f5f680efe6158896c477e061054249fc8fd5fad4d2eb0221b595f22e7dee5c not found: ID does not exist"
Sep 15 21:35:22.573645 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573625    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-envoy\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.573743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573681    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlksh\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-kube-api-access-vlksh\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.573743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573702    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-token\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.573824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573740    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-socket\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.573824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573816    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-credential-socket\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.573924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573843    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-data\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.573924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573910    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-podinfo\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.574028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573963    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istiod-ca-cert\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.574028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.573998    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-certs\") pod \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\" (UID: \"9b90fcf5-05dc-4164-b3fc-4342e91055d3\") "
Sep 15 21:35:22.574133 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574084    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:22.574368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574266    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:22.574368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574334    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:22.574767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574638    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:22.574767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574688    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.574767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574711    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.574767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574727    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.574767 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.574732    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:35:22.576052 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.576026    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-kube-api-access-vlksh" (OuterVolumeSpecName: "kube-api-access-vlksh") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "kube-api-access-vlksh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:35:22.576353 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.576332    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:22.576565 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.576543    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:35:22.576777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.576762    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b90fcf5-05dc-4164-b3fc-4342e91055d3" (UID: "9b90fcf5-05dc-4164-b3fc-4342e91055d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:35:22.675297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.675271    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.675297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.675293    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.675297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.675303    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.675551 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.675313    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.675551 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.675328    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlksh\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-kube-api-access-vlksh\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.675551 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.675341    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b90fcf5-05dc-4164-b3fc-4342e91055d3-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:35:22.865042 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.865007    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"]
Sep 15 21:35:22.869148 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:22.869110    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4e476d59-openshift-default-669cf6f668-mws26"]
Sep 15 21:35:23.769870 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:23.769832    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" path="/var/lib/kubelet/pods/9b90fcf5-05dc-4164-b3fc-4342e91055d3/volumes"
Sep 15 21:35:32.152754 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.152677    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"]
Sep 15 21:35:32.153100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.153065    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy"
Sep 15 21:35:32.153100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.153076    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy"
Sep 15 21:35:32.153100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.153086    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy"
Sep 15 21:35:32.153100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.153094    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy"
Sep 15 21:35:32.153267 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.153163    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy"
Sep 15 21:35:32.157682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.157664    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.161785 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.161769    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a276f9\"/\"embedding-w0-a276f9-kserve-self-signed-certs\""
Sep 15 21:35:32.168523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.168491    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94712580-5511-48e1-9313-6082aa13b707-tls-certs\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.168634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.168546    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-tmp-dir\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.168634 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.168608    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-model-cache\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.168726 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.168644    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-dshm\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.168726 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.168690    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44g6\" (UniqueName: \"kubernetes.io/projected/94712580-5511-48e1-9313-6082aa13b707-kube-api-access-l44g6\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.168810 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.168743    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-home\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.174974 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.174948    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"]
Sep 15 21:35:32.269866 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.269826    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94712580-5511-48e1-9313-6082aa13b707-tls-certs\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.269873    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-tmp-dir\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.269919    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-model-cache\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.269962    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-dshm\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.270020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44g6\" (UniqueName: \"kubernetes.io/projected/94712580-5511-48e1-9313-6082aa13b707-kube-api-access-l44g6\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.270054    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-home\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270780 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.270750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-model-cache\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.270759    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-home\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.270874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.270838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-tmp-dir\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.272554 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.272536    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94712580-5511-48e1-9313-6082aa13b707-tls-certs\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.272628 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.272560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-dshm\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.284477 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.284456    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44g6\" (UniqueName: \"kubernetes.io/projected/94712580-5511-48e1-9313-6082aa13b707-kube-api-access-l44g6\") pod \"embedding-w0-a276f9-kserve-58c66d4585-8bmln\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") " pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.467100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.467064    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:32.752150 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:32.752122    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"]
Sep 15 21:35:32.753101 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:35:32.753079    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94712580_5511_48e1_9313_6082aa13b707.slice/crio-2529e513e4496bac35864c764c400a033b95f15ae8ecdcd4a22e12588bf65de4 WatchSource:0}: Error finding container 2529e513e4496bac35864c764c400a033b95f15ae8ecdcd4a22e12588bf65de4: Status 404 returned error can't find the container with id 2529e513e4496bac35864c764c400a033b95f15ae8ecdcd4a22e12588bf65de4
Sep 15 21:35:33.602396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:33.602365    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln" event={"ID":"94712580-5511-48e1-9313-6082aa13b707","Type":"ContainerStarted","Data":"4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0"}
Sep 15 21:35:33.602396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:33.602401    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln" event={"ID":"94712580-5511-48e1-9313-6082aa13b707","Type":"ContainerStarted","Data":"2529e513e4496bac35864c764c400a033b95f15ae8ecdcd4a22e12588bf65de4"}
Sep 15 21:35:33.622955 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:33.622903    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln" podStartSLOduration=1.622886609 podStartE2EDuration="1.622886609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:33.620946057 +0000 UTC m=+1632.422814999" watchObservedRunningTime="2026-09-15 21:35:33.622886609 +0000 UTC m=+1632.424755551"
Sep 15 21:35:42.467931 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:42.467892    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:42.468396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:42.467960    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:42.480736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:42.480711    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:35:42.668850 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:35:42.668822    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:36:13.426570 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.426535    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"]
Sep 15 21:36:13.427046 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.427023    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b90fcf5-05dc-4164-b3fc-4342e91055d3" containerName="istio-proxy"
Sep 15 21:36:13.430368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.430347    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.432979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.432958    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0753ad-openshift-default-dockercfg-nqvrr\""
Sep 15 21:36:13.445373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.445347    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"]
Sep 15 21:36:13.503256 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503182    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-socket\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503440 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503295    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-credential-socket\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503440 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503335    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istiod-ca-cert\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503440 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl62z\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-kube-api-access-fl62z\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503479    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-data\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503525    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-token\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503592    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-certs\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503620 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503607    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-envoy\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.503773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.503661    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-podinfo\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.604933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.604897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-certs\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.604933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.604931    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-envoy\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.604963    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-podinfo\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.604993    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-socket\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605023    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-credential-socket\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605056    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istiod-ca-cert\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605123    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl62z\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-kube-api-access-fl62z\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605173    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-data\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-token\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605345    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-certs\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605385    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-socket\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.605692 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.605598    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-data\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.606079 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.606055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istiod-ca-cert\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.606184 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.606162    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-credential-socket\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.607537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.607521    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-envoy\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.607664 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.607650    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-podinfo\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.615033 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.615010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-token\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.615749 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.615727    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl62z\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-kube-api-access-fl62z\") pod \"e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") " pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:13.740818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:13.740732    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:14.028555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.028531    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"]
Sep 15 21:36:14.030605 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:36:14.030568    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac62d37_8f52_4ccf_acae_5e56bf196b4c.slice/crio-edc2a7462aa2d79a368321fc77351e4264fbe16bb5db457623690391c8e20ac6 WatchSource:0}: Error finding container edc2a7462aa2d79a368321fc77351e4264fbe16bb5db457623690391c8e20ac6: Status 404 returned error can't find the container with id edc2a7462aa2d79a368321fc77351e4264fbe16bb5db457623690391c8e20ac6
Sep 15 21:36:14.033066 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.033030    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:14.033167 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.033098    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:14.033167 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.033125    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:14.840694 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.840660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" event={"ID":"aac62d37-8f52-4ccf-acae-5e56bf196b4c","Type":"ContainerStarted","Data":"e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"}
Sep 15 21:36:14.841074 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.840708    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" event={"ID":"aac62d37-8f52-4ccf-acae-5e56bf196b4c","Type":"ContainerStarted","Data":"edc2a7462aa2d79a368321fc77351e4264fbe16bb5db457623690391c8e20ac6"}
Sep 15 21:36:14.863722 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:14.863668    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podStartSLOduration=1.863654807 podStartE2EDuration="1.863654807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:14.862334896 +0000 UTC m=+1673.664203858" watchObservedRunningTime="2026-09-15 21:36:14.863654807 +0000 UTC m=+1673.665523750"
Sep 15 21:36:15.741333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:15.741300    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:15.746395 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:15.746372    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:15.846293 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:15.846259    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:15.847318 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:15.847295    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:35.076901 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.076865    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"]
Sep 15 21:36:35.089233 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.089170    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"]
Sep 15 21:36:35.089386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.089296    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:35.091661 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.091636    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:36:35.147167 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.147130    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51738f91-585d-4e71-bf56-d07b75957f02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl\" (UID: \"51738f91-585d-4e71-bf56-d07b75957f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:35.248771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.248736    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51738f91-585d-4e71-bf56-d07b75957f02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl\" (UID: \"51738f91-585d-4e71-bf56-d07b75957f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:35.249443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.249419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51738f91-585d-4e71-bf56-d07b75957f02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl\" (UID: \"51738f91-585d-4e71-bf56-d07b75957f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:35.400163 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.400073    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:35.680418 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.680387    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"]
Sep 15 21:36:35.682246 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:36:35.682213    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51738f91_585d_4e71_bf56_d07b75957f02.slice/crio-a7c652d9c12aab615c296d05ae0c25911c760eea2e6fee6ef637c113bd908805 WatchSource:0}: Error finding container a7c652d9c12aab615c296d05ae0c25911c760eea2e6fee6ef637c113bd908805: Status 404 returned error can't find the container with id a7c652d9c12aab615c296d05ae0c25911c760eea2e6fee6ef637c113bd908805
Sep 15 21:36:35.960590 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:35.960507    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl" event={"ID":"51738f91-585d-4e71-bf56-d07b75957f02","Type":"ContainerStarted","Data":"a7c652d9c12aab615c296d05ae0c25911c760eea2e6fee6ef637c113bd908805"}
Sep 15 21:36:36.968256 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:36.968221    2551 generic.go:356] "Generic (PLEG): container finished" podID="51738f91-585d-4e71-bf56-d07b75957f02" containerID="c2a887faede87b81dd7de8dd25b6768c95d022e82fe6f59ae9697d6b336ab92a" exitCode=0
Sep 15 21:36:36.968663 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:36.968307    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl" event={"ID":"51738f91-585d-4e71-bf56-d07b75957f02","Type":"ContainerDied","Data":"c2a887faede87b81dd7de8dd25b6768c95d022e82fe6f59ae9697d6b336ab92a"}
Sep 15 21:36:38.115619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.115593    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:38.180769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.180736    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51738f91-585d-4e71-bf56-d07b75957f02-maas-api-service-ca\") pod \"51738f91-585d-4e71-bf56-d07b75957f02\" (UID: \"51738f91-585d-4e71-bf56-d07b75957f02\") "
Sep 15 21:36:38.181157 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.181126    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51738f91-585d-4e71-bf56-d07b75957f02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51738f91-585d-4e71-bf56-d07b75957f02" (UID: "51738f91-585d-4e71-bf56-d07b75957f02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:38.282219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.282113    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51738f91-585d-4e71-bf56-d07b75957f02-maas-api-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:38.980980 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.980954    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"
Sep 15 21:36:38.980980 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.980977    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl" event={"ID":"51738f91-585d-4e71-bf56-d07b75957f02","Type":"ContainerDied","Data":"a7c652d9c12aab615c296d05ae0c25911c760eea2e6fee6ef637c113bd908805"}
Sep 15 21:36:38.981205 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:38.981022    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7c652d9c12aab615c296d05ae0c25911c760eea2e6fee6ef637c113bd908805"
Sep 15 21:36:39.936889 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:39.936851    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"]
Sep 15 21:36:39.951155 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:39.951111    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"]
Sep 15 21:36:42.433344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.433304    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"]
Sep 15 21:36:42.433757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.433729    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51738f91-585d-4e71-bf56-d07b75957f02" containerName="revoke-keys"
Sep 15 21:36:42.433757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.433743    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="51738f91-585d-4e71-bf56-d07b75957f02" containerName="revoke-keys"
Sep 15 21:36:42.433860 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.433851    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="51738f91-585d-4e71-bf56-d07b75957f02" containerName="revoke-keys"
Sep 15 21:36:42.437020 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.437001    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.439249 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.439227    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-30e32e-openshift-default-dockercfg-4lbg4\""
Sep 15 21:36:42.455148 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.455121    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"]
Sep 15 21:36:42.530986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.530946    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d547ba7-f53f-4638-9960-746741734ddf-istio-podinfo\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.531303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.531280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99x9l\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-kube-api-access-99x9l\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.531509 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.531493    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-credential-socket\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.531627 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.531611    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d547ba7-f53f-4638-9960-746741734ddf-istiod-ca-cert\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.531743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.531729    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-envoy\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.532011 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.531993    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-socket\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.532175 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.532159    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-data\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.532326 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.532310    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-certs\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.532443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.532430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-istio-token\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-socket\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633512 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633493    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-data\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633528    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-certs\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633568    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-istio-token\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633621    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d547ba7-f53f-4638-9960-746741734ddf-istio-podinfo\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633646    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99x9l\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-kube-api-access-99x9l\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-credential-socket\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.633779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d547ba7-f53f-4638-9960-746741734ddf-istiod-ca-cert\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.634185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633795    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-envoy\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.634185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.633933    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-data\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.634185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.634174    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-certs\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.634484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.634459    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-credential-socket\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.634568 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.634519    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-socket\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.634828 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.634805    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d547ba7-f53f-4638-9960-746741734ddf-istiod-ca-cert\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.636433 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.636412    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d547ba7-f53f-4638-9960-746741734ddf-istio-podinfo\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.636691 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.636674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-envoy\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.645568 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.645537    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-istio-token\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.645687 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.645621    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99x9l\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-kube-api-access-99x9l\") pod \"e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") " pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:42.747592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:42.747494    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:43.031409 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:43.031382    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"]
Sep 15 21:36:43.034262 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:36:43.034225    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d547ba7_f53f_4638_9960_746741734ddf.slice/crio-6cae71bee51ddd1c931cc2a47f7e741f947d08420ca7d9a48d7d1263a043663c WatchSource:0}: Error finding container 6cae71bee51ddd1c931cc2a47f7e741f947d08420ca7d9a48d7d1263a043663c: Status 404 returned error can't find the container with id 6cae71bee51ddd1c931cc2a47f7e741f947d08420ca7d9a48d7d1263a043663c
Sep 15 21:36:43.036992 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:43.036950    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:43.037088 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:43.037026    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:43.037088 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:43.037057    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:44.011263 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:44.011221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" event={"ID":"9d547ba7-f53f-4638-9960-746741734ddf","Type":"ContainerStarted","Data":"95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"}
Sep 15 21:36:44.011761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:44.011331    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" event={"ID":"9d547ba7-f53f-4638-9960-746741734ddf","Type":"ContainerStarted","Data":"6cae71bee51ddd1c931cc2a47f7e741f947d08420ca7d9a48d7d1263a043663c"}
Sep 15 21:36:44.067983 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:44.067918    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podStartSLOduration=2.067895423 podStartE2EDuration="2.067895423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:44.062477798 +0000 UTC m=+1702.864346737" watchObservedRunningTime="2026-09-15 21:36:44.067895423 +0000 UTC m=+1702.869764367"
Sep 15 21:36:44.748757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:44.748714    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:44.754037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:44.754012    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:45.017277 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:45.017175    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:45.018133 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:45.018116    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:36:53.075626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:53.075533    2551 generic.go:356] "Generic (PLEG): container finished" podID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerID="e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73" exitCode=0
Sep 15 21:36:53.075626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:53.075608    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" event={"ID":"aac62d37-8f52-4ccf-acae-5e56bf196b4c","Type":"ContainerDied","Data":"e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"}
Sep 15 21:36:53.076104 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:53.076053    2551 scope.go:117] "RemoveContainer" containerID="e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"
Sep 15 21:36:53.077475 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:53.077443    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:53.077585 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:53.077514    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:53.077585 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:53.077548    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:36:54.093954 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:54.093915    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" event={"ID":"aac62d37-8f52-4ccf-acae-5e56bf196b4c","Type":"ContainerStarted","Data":"40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367"}
Sep 15 21:36:54.741651 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:54.741621    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:54.741886 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:54.741863    2551 patch_prober.go:28] interesting pod/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 21:36:54.741950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:54.741927    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 21:36:54.968834 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:54.968788    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm" podUID="882a2d76-cecd-49d9-b092-b777907ce81f" containerName="main" containerID="cri-o://d8b7ad4341f6e92ee1b47af37c0a1840a7586855b79dc8c7fafe4f7010d8a35f" gracePeriod=45
Sep 15 21:36:54.981653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:54.981578    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln" podUID="94712580-5511-48e1-9313-6082aa13b707" containerName="main" containerID="cri-o://4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0" gracePeriod=45
Sep 15 21:36:55.025147 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:36:55.025110    2551 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod882a2d76_cecd_49d9_b092_b777907ce81f.slice/crio-d8b7ad4341f6e92ee1b47af37c0a1840a7586855b79dc8c7fafe4f7010d8a35f.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 21:36:55.103069 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.103035    2551 generic.go:356] "Generic (PLEG): container finished" podID="882a2d76-cecd-49d9-b092-b777907ce81f" containerID="d8b7ad4341f6e92ee1b47af37c0a1840a7586855b79dc8c7fafe4f7010d8a35f" exitCode=0
Sep 15 21:36:55.103524 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.103125    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm" event={"ID":"882a2d76-cecd-49d9-b092-b777907ce81f","Type":"ContainerDied","Data":"d8b7ad4341f6e92ee1b47af37c0a1840a7586855b79dc8c7fafe4f7010d8a35f"}
Sep 15 21:36:55.255445 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.255376    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:36:55.324028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.323731    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:36:55.408377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408349    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882a2d76-cecd-49d9-b092-b777907ce81f-tls-certs\") pod \"882a2d76-cecd-49d9-b092-b777907ce81f\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") "
Sep 15 21:36:55.408516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408432    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq2df\" (UniqueName: \"kubernetes.io/projected/882a2d76-cecd-49d9-b092-b777907ce81f-kube-api-access-vq2df\") pod \"882a2d76-cecd-49d9-b092-b777907ce81f\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") "
Sep 15 21:36:55.408516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408474    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-dshm\") pod \"882a2d76-cecd-49d9-b092-b777907ce81f\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") "
Sep 15 21:36:55.408516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408497    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-model-cache\") pod \"882a2d76-cecd-49d9-b092-b777907ce81f\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") "
Sep 15 21:36:55.408635 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408518    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-tmp-dir\") pod \"882a2d76-cecd-49d9-b092-b777907ce81f\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") "
Sep 15 21:36:55.408635 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408561    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-home\") pod \"882a2d76-cecd-49d9-b092-b777907ce81f\" (UID: \"882a2d76-cecd-49d9-b092-b777907ce81f\") "
Sep 15 21:36:55.409029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.408994    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-home" (OuterVolumeSpecName: "home") pod "882a2d76-cecd-49d9-b092-b777907ce81f" (UID: "882a2d76-cecd-49d9-b092-b777907ce81f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.409207 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.409152    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-model-cache" (OuterVolumeSpecName: "model-cache") pod "882a2d76-cecd-49d9-b092-b777907ce81f" (UID: "882a2d76-cecd-49d9-b092-b777907ce81f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.409459 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.409363    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "882a2d76-cecd-49d9-b092-b777907ce81f" (UID: "882a2d76-cecd-49d9-b092-b777907ce81f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.411012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.410985    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-dshm" (OuterVolumeSpecName: "dshm") pod "882a2d76-cecd-49d9-b092-b777907ce81f" (UID: "882a2d76-cecd-49d9-b092-b777907ce81f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.411114 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.411032    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882a2d76-cecd-49d9-b092-b777907ce81f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "882a2d76-cecd-49d9-b092-b777907ce81f" (UID: "882a2d76-cecd-49d9-b092-b777907ce81f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:55.411378 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.411361    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882a2d76-cecd-49d9-b092-b777907ce81f-kube-api-access-vq2df" (OuterVolumeSpecName: "kube-api-access-vq2df") pod "882a2d76-cecd-49d9-b092-b777907ce81f" (UID: "882a2d76-cecd-49d9-b092-b777907ce81f"). InnerVolumeSpecName "kube-api-access-vq2df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:55.509584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509491    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l44g6\" (UniqueName: \"kubernetes.io/projected/94712580-5511-48e1-9313-6082aa13b707-kube-api-access-l44g6\") pod \"94712580-5511-48e1-9313-6082aa13b707\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") "
Sep 15 21:36:55.509740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509614    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-home\") pod \"94712580-5511-48e1-9313-6082aa13b707\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") "
Sep 15 21:36:55.509740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509651    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-tmp-dir\") pod \"94712580-5511-48e1-9313-6082aa13b707\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") "
Sep 15 21:36:55.509740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509675    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-dshm\") pod \"94712580-5511-48e1-9313-6082aa13b707\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") "
Sep 15 21:36:55.509740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509706    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-model-cache\") pod \"94712580-5511-48e1-9313-6082aa13b707\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") "
Sep 15 21:36:55.509968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509793    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94712580-5511-48e1-9313-6082aa13b707-tls-certs\") pod \"94712580-5511-48e1-9313-6082aa13b707\" (UID: \"94712580-5511-48e1-9313-6082aa13b707\") "
Sep 15 21:36:55.509968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509877    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-home" (OuterVolumeSpecName: "home") pod "94712580-5511-48e1-9313-6082aa13b707" (UID: "94712580-5511-48e1-9313-6082aa13b707"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.509968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509885    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94712580-5511-48e1-9313-6082aa13b707" (UID: "94712580-5511-48e1-9313-6082aa13b707"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.509968 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.509938    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-model-cache" (OuterVolumeSpecName: "model-cache") pod "94712580-5511-48e1-9313-6082aa13b707" (UID: "94712580-5511-48e1-9313-6082aa13b707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.510228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510184    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510223    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510234    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510242    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882a2d76-cecd-49d9-b092-b777907ce81f-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510250    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq2df\" (UniqueName: \"kubernetes.io/projected/882a2d76-cecd-49d9-b092-b777907ce81f-kube-api-access-vq2df\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510258    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510267    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510278    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.510330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.510290    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882a2d76-cecd-49d9-b092-b777907ce81f-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.511881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.511859    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94712580-5511-48e1-9313-6082aa13b707-kube-api-access-l44g6" (OuterVolumeSpecName: "kube-api-access-l44g6") pod "94712580-5511-48e1-9313-6082aa13b707" (UID: "94712580-5511-48e1-9313-6082aa13b707"). InnerVolumeSpecName "kube-api-access-l44g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:55.512419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.512397    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-dshm" (OuterVolumeSpecName: "dshm") pod "94712580-5511-48e1-9313-6082aa13b707" (UID: "94712580-5511-48e1-9313-6082aa13b707"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:55.512474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.512398    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94712580-5511-48e1-9313-6082aa13b707-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94712580-5511-48e1-9313-6082aa13b707" (UID: "94712580-5511-48e1-9313-6082aa13b707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:55.611529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.611490    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94712580-5511-48e1-9313-6082aa13b707-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.611529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.611524    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44g6\" (UniqueName: \"kubernetes.io/projected/94712580-5511-48e1-9313-6082aa13b707-kube-api-access-l44g6\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.611529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.611534    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94712580-5511-48e1-9313-6082aa13b707-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:36:55.741121 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.741080    2551 patch_prober.go:28] interesting pod/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 21:36:55.741321 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:55.741143    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 21:36:56.109588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.109501    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"
Sep 15 21:36:56.109588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.109505    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm" event={"ID":"882a2d76-cecd-49d9-b092-b777907ce81f","Type":"ContainerDied","Data":"12b1112a66557f137901130d2c29a1587f46d6a8c32a2388aa332f539161984b"}
Sep 15 21:36:56.109588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.109561    2551 scope.go:117] "RemoveContainer" containerID="d8b7ad4341f6e92ee1b47af37c0a1840a7586855b79dc8c7fafe4f7010d8a35f"
Sep 15 21:36:56.110996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.110884    2551 generic.go:356] "Generic (PLEG): container finished" podID="94712580-5511-48e1-9313-6082aa13b707" containerID="4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0" exitCode=0
Sep 15 21:36:56.110996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.110925    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln" event={"ID":"94712580-5511-48e1-9313-6082aa13b707","Type":"ContainerDied","Data":"4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0"}
Sep 15 21:36:56.110996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.110933    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"
Sep 15 21:36:56.110996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.110954    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln" event={"ID":"94712580-5511-48e1-9313-6082aa13b707","Type":"ContainerDied","Data":"2529e513e4496bac35864c764c400a033b95f15ae8ecdcd4a22e12588bf65de4"}
Sep 15 21:36:56.119211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.119166    2551 scope.go:117] "RemoveContainer" containerID="4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0"
Sep 15 21:36:56.127584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.127569    2551 scope.go:117] "RemoveContainer" containerID="4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0"
Sep 15 21:36:56.127828 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:36:56.127812    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0\": container with ID starting with 4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0 not found: ID does not exist" containerID="4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0"
Sep 15 21:36:56.127877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.127837    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0"} err="failed to get container status \"4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0\": rpc error: code = NotFound desc = could not find container \"4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0\": container with ID starting with 4b074a678a7471ce5c898dddd1b6ed15b7addedd2a552fb7f3a72e21ac5df6c0 not found: ID does not exist"
Sep 15 21:36:56.148290 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.148261    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"]
Sep 15 21:36:56.156273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.156247    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/embedding-w0-a276f9-kserve-58c66d4585-8bmln"]
Sep 15 21:36:56.175528 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.175490    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"]
Sep 15 21:36:56.183768 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.183741    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a276f9/distinct-2-w0-a276f9-kserve-5fb6cc959f-9d8jm"]
Sep 15 21:36:56.741786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.741753    2551 patch_prober.go:28] interesting pod/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 21:36:56.741985 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:56.741809    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 21:36:57.527306 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.527277    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"]
Sep 15 21:36:57.533279 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.533253    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a276f9-pp2kl"]
Sep 15 21:36:57.741376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.741339    2551 patch_prober.go:28] interesting pod/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 21:36:57.741561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.741397    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 21:36:57.792332 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.791454    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51738f91-585d-4e71-bf56-d07b75957f02" path="/var/lib/kubelet/pods/51738f91-585d-4e71-bf56-d07b75957f02/volumes"
Sep 15 21:36:57.792969 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.792937    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="882a2d76-cecd-49d9-b092-b777907ce81f" path="/var/lib/kubelet/pods/882a2d76-cecd-49d9-b092-b777907ce81f/volumes"
Sep 15 21:36:57.793603 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:57.793582    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94712580-5511-48e1-9313-6082aa13b707" path="/var/lib/kubelet/pods/94712580-5511-48e1-9313-6082aa13b707/volumes"
Sep 15 21:36:58.741711 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:58.741672    2551 patch_prober.go:28] interesting pod/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 21:36:58.742112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:58.741728    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 21:36:59.745906 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:59.745877    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:59.746330 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:59.746148    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:36:59.747002 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:36:59.746986    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:37:04.228007 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:04.227972    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"]
Sep 15 21:37:04.228468 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:04.228277    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy" containerID="cri-o://44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922" gracePeriod=30
Sep 15 21:37:09.498439 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.498373    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:37:09.561402 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561374    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-data\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561415    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-token\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561461    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz7n2\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-kube-api-access-gz7n2\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561485    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-socket\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561500    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-envoy\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561518    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-credential-socket\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561602 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561536    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istiod-ca-cert\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.561909 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561693    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:09.562387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561828    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:09.562387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.562154    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-certs\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.562559 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.562457    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-podinfo\") pod \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\" (UID: \"5f77b46f-ea22-43f8-8105-c25ea9cabf4a\") "
Sep 15 21:37:09.562849 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.562825    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:09.562986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.561911    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:09.563137 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.562175    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:09.563137 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.563078    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.563356 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.563168    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.563356 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.563219    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.564219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.564127    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:09.564329 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.564280    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-kube-api-access-gz7n2" (OuterVolumeSpecName: "kube-api-access-gz7n2") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "kube-api-access-gz7n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:09.564396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.564358    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:09.564842 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.564815    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f77b46f-ea22-43f8-8105-c25ea9cabf4a" (UID: "5f77b46f-ea22-43f8-8105-c25ea9cabf4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:37:09.664640 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.664604    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.664640 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.664636    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7n2\" (UniqueName: \"kubernetes.io/projected/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-kube-api-access-gz7n2\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.664640 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.664646    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.664883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.664654    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.664883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.664665    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:09.664883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:09.664673    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f77b46f-ea22-43f8-8105-c25ea9cabf4a-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:10.200019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.199985    2551 generic.go:356] "Generic (PLEG): container finished" podID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerID="44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922" exitCode=0
Sep 15 21:37:10.200236 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.200032    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" event={"ID":"5f77b46f-ea22-43f8-8105-c25ea9cabf4a","Type":"ContainerDied","Data":"44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922"}
Sep 15 21:37:10.200236 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.200046    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"
Sep 15 21:37:10.200236 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.200091    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg" event={"ID":"5f77b46f-ea22-43f8-8105-c25ea9cabf4a","Type":"ContainerDied","Data":"82a29f48a556dce06def72405923ad684f7d5008d8a3b51ecc0c01ad59af1eae"}
Sep 15 21:37:10.200236 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.200114    2551 scope.go:117] "RemoveContainer" containerID="44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922"
Sep 15 21:37:10.209311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.209255    2551 scope.go:117] "RemoveContainer" containerID="2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"
Sep 15 21:37:10.218206 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.218167    2551 scope.go:117] "RemoveContainer" containerID="44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922"
Sep 15 21:37:10.218572 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:37:10.218552    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922\": container with ID starting with 44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922 not found: ID does not exist" containerID="44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922"
Sep 15 21:37:10.218633 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.218579    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922"} err="failed to get container status \"44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922\": rpc error: code = NotFound desc = could not find container \"44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922\": container with ID starting with 44d4ea9f16e79c6985b979ecb40b543190924d2b25de69d8875741654afb9922 not found: ID does not exist"
Sep 15 21:37:10.218633 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.218597    2551 scope.go:117] "RemoveContainer" containerID="2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"
Sep 15 21:37:10.218773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.218756    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"]
Sep 15 21:37:10.218835 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:37:10.218822    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7\": container with ID starting with 2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7 not found: ID does not exist" containerID="2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"
Sep 15 21:37:10.218875 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.218838    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7"} err="failed to get container status \"2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7\": rpc error: code = NotFound desc = could not find container \"2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7\": container with ID starting with 2924868bb6f92ad561571759a5b90957a770b0f022ab2cf5b5a047cb85c9d2b7 not found: ID does not exist"
Sep 15 21:37:10.222307 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:10.222285    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a276f9-openshift-default-655fb6b746-zdldg"]
Sep 15 21:37:11.776931 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:11.776888    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" path="/var/lib/kubelet/pods/5f77b46f-ea22-43f8-8105-c25ea9cabf4a/volumes"
Sep 15 21:37:18.249421 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:18.249331    2551 generic.go:356] "Generic (PLEG): container finished" podID="9d547ba7-f53f-4638-9960-746741734ddf" containerID="95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e" exitCode=0
Sep 15 21:37:18.249941 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:18.249410    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" event={"ID":"9d547ba7-f53f-4638-9960-746741734ddf","Type":"ContainerDied","Data":"95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"}
Sep 15 21:37:18.250010 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:18.249961    2551 scope.go:117] "RemoveContainer" containerID="95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"
Sep 15 21:37:18.251251 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:18.251221    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:37:18.251326 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:18.251291    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:37:18.251363 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:18.251326    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:37:19.256701 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:19.256665    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" event={"ID":"9d547ba7-f53f-4638-9960-746741734ddf","Type":"ContainerStarted","Data":"482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097"}
Sep 15 21:37:19.748720 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:19.748683    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:37:19.748903 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:19.748875    2551 patch_prober.go:28] interesting pod/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 21:37:19.748955 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:19.748925    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 21:37:20.748529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:20.748486    2551 patch_prober.go:28] interesting pod/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 21:37:20.748989 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:20.748549    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 21:37:21.748667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:21.748632    2551 patch_prober.go:28] interesting pod/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 21:37:21.749048 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:21.748686    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 21:37:21.909881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:21.909852    2551 scope.go:117] "RemoveContainer" containerID="b0ca3c92ed0dd0eb36a8023f91b8603e6fe5578e6069106ae02546fc89788dc9"
Sep 15 21:37:22.748431 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:22.748394    2551 patch_prober.go:28] interesting pod/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 21:37:22.748616 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:22.748467    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 21:37:23.748113 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:23.748068    2551 patch_prober.go:28] interesting pod/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 21:37:23.748660 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:23.748156    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 21:37:24.752254 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:24.752223    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:37:24.752753 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:24.752603    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:37:24.753249 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:24.753231    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:37:32.691938 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:32.691901    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"]
Sep 15 21:37:32.692410 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:32.692259    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy" containerID="cri-o://40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367" gracePeriod=30
Sep 15 21:37:37.964441 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:37.964373    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:37:38.076930 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.076891    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-envoy\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.076956    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-credential-socket\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.076982    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istiod-ca-cert\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077065    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl62z\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-kube-api-access-fl62z\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077107 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077088    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-socket\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077112    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-podinfo\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077146    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-data\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077241    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-certs\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077288    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-token\") pod \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\" (UID: \"aac62d37-8f52-4ccf-acae-5e56bf196b4c\") "
Sep 15 21:37:38.077355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077315    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:38.077616 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077536    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:38.077616 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077557    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:38.077616 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077567    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:38.077616 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077585    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:37:38.077947 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077924    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.078037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077949    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.078037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077964    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.078037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077977    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.078037 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.077991    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.079347 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.079319    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:37:38.079524 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.079503    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-kube-api-access-fl62z" (OuterVolumeSpecName: "kube-api-access-fl62z") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "kube-api-access-fl62z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:38.079601 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.079535    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:37:38.079743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.079728    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "aac62d37-8f52-4ccf-acae-5e56bf196b4c" (UID: "aac62d37-8f52-4ccf-acae-5e56bf196b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:37:38.179443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.179415    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl62z\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-kube-api-access-fl62z\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.179443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.179442    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.179443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.179450    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.179443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.179458    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac62d37-8f52-4ccf-acae-5e56bf196b4c-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:37:38.366396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.366303    2551 generic.go:356] "Generic (PLEG): container finished" podID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerID="40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367" exitCode=0
Sep 15 21:37:38.366396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.366346    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"
Sep 15 21:37:38.366592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.366392    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" event={"ID":"aac62d37-8f52-4ccf-acae-5e56bf196b4c","Type":"ContainerDied","Data":"40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367"}
Sep 15 21:37:38.366592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.366446    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj" event={"ID":"aac62d37-8f52-4ccf-acae-5e56bf196b4c","Type":"ContainerDied","Data":"edc2a7462aa2d79a368321fc77351e4264fbe16bb5db457623690391c8e20ac6"}
Sep 15 21:37:38.366592 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.366472    2551 scope.go:117] "RemoveContainer" containerID="40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367"
Sep 15 21:37:38.376255 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.376234    2551 scope.go:117] "RemoveContainer" containerID="e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"
Sep 15 21:37:38.384933 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.384914    2551 scope.go:117] "RemoveContainer" containerID="40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367"
Sep 15 21:37:38.385314 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:37:38.385295    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367\": container with ID starting with 40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367 not found: ID does not exist" containerID="40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367"
Sep 15 21:37:38.385376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.385323    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367"} err="failed to get container status \"40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367\": rpc error: code = NotFound desc = could not find container \"40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367\": container with ID starting with 40995fa98f945bb152ec37380e274a7a27fc62c76e980919d24c775ca6da3367 not found: ID does not exist"
Sep 15 21:37:38.385376 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.385341    2551 scope.go:117] "RemoveContainer" containerID="e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"
Sep 15 21:37:38.385658 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:37:38.385636    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73\": container with ID starting with e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73 not found: ID does not exist" containerID="e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"
Sep 15 21:37:38.385720 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.385669    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73"} err="failed to get container status \"e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73\": rpc error: code = NotFound desc = could not find container \"e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73\": container with ID starting with e382bc9b80eb2a9cd2db458d09d9a9d8a8874310497c20601e8f28043c554a73 not found: ID does not exist"
Sep 15 21:37:38.388117 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.388096    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"]
Sep 15 21:37:38.394303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:38.394281    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0753ad-openshift-default-7446dbd7cb-z75hj"]
Sep 15 21:37:39.769308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:39.769276    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" path="/var/lib/kubelet/pods/aac62d37-8f52-4ccf-acae-5e56bf196b4c/volumes"
Sep 15 21:37:55.855886 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:55.855852    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"]
Sep 15 21:37:55.856499 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:37:55.856065    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy" containerID="cri-o://482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097" gracePeriod=30
Sep 15 21:38:01.115685 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.115653    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:38:01.223264 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223234    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-istio-token\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223274    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d547ba7-f53f-4638-9960-746741734ddf-istio-podinfo\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223325    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-data\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223380    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99x9l\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-kube-api-access-99x9l\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223435 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223428    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d547ba7-f53f-4638-9960-746741734ddf-istiod-ca-cert\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223460    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-credential-socket\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223492    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-certs\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223551    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-socket\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223605    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-envoy\") pod \"9d547ba7-f53f-4638-9960-746741734ddf\" (UID: \"9d547ba7-f53f-4638-9960-746741734ddf\") "
Sep 15 21:38:01.223881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223689    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:01.223881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223739    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:01.223881 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223838    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:01.224019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223945    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:01.224019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.223971    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d547ba7-f53f-4638-9960-746741734ddf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:01.224179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.224104    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d547ba7-f53f-4638-9960-746741734ddf-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.224456 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.224305    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.224456 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.224321    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.224456 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.224334    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.224456 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.224347    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.225847 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.225807    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-kube-api-access-99x9l" (OuterVolumeSpecName: "kube-api-access-99x9l") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "kube-api-access-99x9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:01.225973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.225850    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d547ba7-f53f-4638-9960-746741734ddf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:01.225973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.225849    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:01.225973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.225879    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d547ba7-f53f-4638-9960-746741734ddf" (UID: "9d547ba7-f53f-4638-9960-746741734ddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:01.324884 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.324845    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.324884 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.324876    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d547ba7-f53f-4638-9960-746741734ddf-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.325085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.324891    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99x9l\" (UniqueName: \"kubernetes.io/projected/9d547ba7-f53f-4638-9960-746741734ddf-kube-api-access-99x9l\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.325085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.324906    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d547ba7-f53f-4638-9960-746741734ddf-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:01.491970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.491867    2551 generic.go:356] "Generic (PLEG): container finished" podID="9d547ba7-f53f-4638-9960-746741734ddf" containerID="482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097" exitCode=0
Sep 15 21:38:01.491970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.491931    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"
Sep 15 21:38:01.491970 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.491953    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" event={"ID":"9d547ba7-f53f-4638-9960-746741734ddf","Type":"ContainerDied","Data":"482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097"}
Sep 15 21:38:01.492289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.492008    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm" event={"ID":"9d547ba7-f53f-4638-9960-746741734ddf","Type":"ContainerDied","Data":"6cae71bee51ddd1c931cc2a47f7e741f947d08420ca7d9a48d7d1263a043663c"}
Sep 15 21:38:01.492289 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.492030    2551 scope.go:117] "RemoveContainer" containerID="482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097"
Sep 15 21:38:01.502388 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.502367    2551 scope.go:117] "RemoveContainer" containerID="95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"
Sep 15 21:38:01.510862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.510848    2551 scope.go:117] "RemoveContainer" containerID="482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097"
Sep 15 21:38:01.511110 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:01.511089    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097\": container with ID starting with 482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097 not found: ID does not exist" containerID="482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097"
Sep 15 21:38:01.511160 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.511118    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097"} err="failed to get container status \"482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097\": rpc error: code = NotFound desc = could not find container \"482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097\": container with ID starting with 482cd5cecb06abbb348db04aa3853cd5475dc57ae6b5a4b852c79bdef099a097 not found: ID does not exist"
Sep 15 21:38:01.511160 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.511132    2551 scope.go:117] "RemoveContainer" containerID="95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"
Sep 15 21:38:01.511388 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:01.511370    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e\": container with ID starting with 95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e not found: ID does not exist" containerID="95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"
Sep 15 21:38:01.511429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.511398    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e"} err="failed to get container status \"95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e\": rpc error: code = NotFound desc = could not find container \"95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e\": container with ID starting with 95fc86d30b00571f5c456861ea626b978d7d0c4d25bf826fccfb61e99545df3e not found: ID does not exist"
Sep 15 21:38:01.516179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.516155    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"]
Sep 15 21:38:01.519829 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.519807    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-30e32e-openshift-default-7576f64bd6-lgvqm"]
Sep 15 21:38:01.769129 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:01.769051    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d547ba7-f53f-4638-9960-746741734ddf" path="/var/lib/kubelet/pods/9d547ba7-f53f-4638-9960-746741734ddf/volumes"
Sep 15 21:38:07.675348 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:07.675311    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"]
Sep 15 21:38:07.697127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:07.697080    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"]
Sep 15 21:38:15.966310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:15.966223    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"]
Sep 15 21:38:15.966730 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:15.966453    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy" containerID="cri-o://4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57" gracePeriod=30
Sep 15 21:38:18.843414 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:18.843374    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"]
Sep 15 21:38:18.845932 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:18.845912    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-625609-29825130-9zqkt"]
Sep 15 21:38:19.769485 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:19.769450    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7e424e-cc9f-48b8-95cc-8715e46d07fb" path="/var/lib/kubelet/pods/4b7e424e-cc9f-48b8-95cc-8715e46d07fb/volumes"
Sep 15 21:38:21.234288 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.234217    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:38:21.324621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324582    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-socket\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.324808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324659    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-data\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.324808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324702    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istiod-ca-cert\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.324808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324775    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-credential-socket\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.324979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324818    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-certs\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.324979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324865    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:21.324979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324942    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-podinfo\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.324979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324967    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:21.325229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.324993    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-token\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.325229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325077    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:21.325229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325062    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:21.325229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325165    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-envoy\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.325403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325234    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swdtm\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-kube-api-access-swdtm\") pod \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\" (UID: \"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1\") "
Sep 15 21:38:21.325403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325230    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:21.325619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325598    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.325619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325619    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.325800 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325632    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.325800 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325646    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.325800 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.325659    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.327224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.327178    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:21.327224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.327183    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:21.327816 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.327785    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:21.327816 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.327797    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-kube-api-access-swdtm" (OuterVolumeSpecName: "kube-api-access-swdtm") pod "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" (UID: "9680b161-7c0c-4fb1-8784-66a6a0ccd9d1"). InnerVolumeSpecName "kube-api-access-swdtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:21.426219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.426160    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.426219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.426219    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.426415 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.426230    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.426415 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.426239    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdtm\" (UniqueName: \"kubernetes.io/projected/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1-kube-api-access-swdtm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:21.598966 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.598879    2551 generic.go:356] "Generic (PLEG): container finished" podID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerID="4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57" exitCode=0
Sep 15 21:38:21.598966 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.598936    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"
Sep 15 21:38:21.599143 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.598958    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" event={"ID":"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1","Type":"ContainerDied","Data":"4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57"}
Sep 15 21:38:21.599143 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.599013    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv" event={"ID":"9680b161-7c0c-4fb1-8784-66a6a0ccd9d1","Type":"ContainerDied","Data":"48096d8ab4d9309c8fc4ccbefd5a726ffbb7ded77b9cf360f9cb23e64207308c"}
Sep 15 21:38:21.599143 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.599037    2551 scope.go:117] "RemoveContainer" containerID="4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57"
Sep 15 21:38:21.609583 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.609559    2551 scope.go:117] "RemoveContainer" containerID="ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"
Sep 15 21:38:21.618736 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.618710    2551 scope.go:117] "RemoveContainer" containerID="4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57"
Sep 15 21:38:21.619053 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:21.619032    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57\": container with ID starting with 4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57 not found: ID does not exist" containerID="4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57"
Sep 15 21:38:21.619144 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.619064    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57"} err="failed to get container status \"4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57\": rpc error: code = NotFound desc = could not find container \"4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57\": container with ID starting with 4d4313356e10890ac51f1beda52b55b59ce223ec0f50c531fbd46cabdf2b4a57 not found: ID does not exist"
Sep 15 21:38:21.619144 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.619091    2551 scope.go:117] "RemoveContainer" containerID="ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"
Sep 15 21:38:21.619378 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:21.619355    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf\": container with ID starting with ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf not found: ID does not exist" containerID="ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"
Sep 15 21:38:21.619429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.619385    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf"} err="failed to get container status \"ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf\": rpc error: code = NotFound desc = could not find container \"ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf\": container with ID starting with ac89a13fd4f0092a0e327d1dfee81ec4cb4826c8c3c4918d119dccd191d967cf not found: ID does not exist"
Sep 15 21:38:21.623614 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.623594    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"]
Sep 15 21:38:21.632783 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.632759    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a3aa55-openshift-default-5d9fb5ddd6-gk7xv"]
Sep 15 21:38:21.773233 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.772587    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" path="/var/lib/kubelet/pods/9680b161-7c0c-4fb1-8784-66a6a0ccd9d1/volumes"
Sep 15 21:38:21.941004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:21.940979    2551 scope.go:117] "RemoveContainer" containerID="9813c51c58a81012402b5ece04e5fca295bd15d0e735e57853734dbbb224d1bc"
Sep 15 21:38:22.695312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:22.695274    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt" podUID="ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" containerName="main" containerID="cri-o://1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115" gracePeriod=45
Sep 15 21:38:22.721451 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:22.721416    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6" podUID="76f7a4da-4b35-47e5-b761-831212f95f0b" containerName="main" containerID="cri-o://bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c" gracePeriod=45
Sep 15 21:38:22.986574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:22.986544    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:38:23.009896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.009868    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:38:23.144659 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144623    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-dshm\") pod \"76f7a4da-4b35-47e5-b761-831212f95f0b\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") "
Sep 15 21:38:23.144864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144677    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-home\") pod \"76f7a4da-4b35-47e5-b761-831212f95f0b\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") "
Sep 15 21:38:23.144864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144713    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdxn5\" (UniqueName: \"kubernetes.io/projected/76f7a4da-4b35-47e5-b761-831212f95f0b-kube-api-access-fdxn5\") pod \"76f7a4da-4b35-47e5-b761-831212f95f0b\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") "
Sep 15 21:38:23.144864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144743    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-model-cache\") pod \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") "
Sep 15 21:38:23.144864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144802    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7a4da-4b35-47e5-b761-831212f95f0b-tls-certs\") pod \"76f7a4da-4b35-47e5-b761-831212f95f0b\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") "
Sep 15 21:38:23.144864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144834    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7rlw\" (UniqueName: \"kubernetes.io/projected/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-kube-api-access-x7rlw\") pod \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") "
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144869    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tmp-dir\") pod \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") "
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144891    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-home\") pod \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") "
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144922    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tls-certs\") pod \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") "
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144961    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-model-cache\") pod \"76f7a4da-4b35-47e5-b761-831212f95f0b\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") "
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.144990    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-home" (OuterVolumeSpecName: "home") pod "76f7a4da-4b35-47e5-b761-831212f95f0b" (UID: "76f7a4da-4b35-47e5-b761-831212f95f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145034    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-dshm\") pod \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\" (UID: \"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f\") "
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145037    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" (UID: "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.145128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145075    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-tmp-dir\") pod \"76f7a4da-4b35-47e5-b761-831212f95f0b\" (UID: \"76f7a4da-4b35-47e5-b761-831212f95f0b\") "
Sep 15 21:38:23.145580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145236    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-home" (OuterVolumeSpecName: "home") pod "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" (UID: "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.145580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145438    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76f7a4da-4b35-47e5-b761-831212f95f0b" (UID: "76f7a4da-4b35-47e5-b761-831212f95f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.145690 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145645    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "76f7a4da-4b35-47e5-b761-831212f95f0b" (UID: "76f7a4da-4b35-47e5-b761-831212f95f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.145750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145687    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.145750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145640    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" (UID: "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.145872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145707    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.145872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145812    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.145872 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.145827    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.147238 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.147174    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f7a4da-4b35-47e5-b761-831212f95f0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76f7a4da-4b35-47e5-b761-831212f95f0b" (UID: "76f7a4da-4b35-47e5-b761-831212f95f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:23.147724 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.147696    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f7a4da-4b35-47e5-b761-831212f95f0b-kube-api-access-fdxn5" (OuterVolumeSpecName: "kube-api-access-fdxn5") pod "76f7a4da-4b35-47e5-b761-831212f95f0b" (UID: "76f7a4da-4b35-47e5-b761-831212f95f0b"). InnerVolumeSpecName "kube-api-access-fdxn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:23.147724 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.147707    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-dshm" (OuterVolumeSpecName: "dshm") pod "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" (UID: "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.147853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.147806    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-dshm" (OuterVolumeSpecName: "dshm") pod "76f7a4da-4b35-47e5-b761-831212f95f0b" (UID: "76f7a4da-4b35-47e5-b761-831212f95f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:23.147853 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.147811    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-kube-api-access-x7rlw" (OuterVolumeSpecName: "kube-api-access-x7rlw") pod "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" (UID: "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f"). InnerVolumeSpecName "kube-api-access-x7rlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:23.147959 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.147944    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" (UID: "ec29a1f6-8d6a-4112-a1ff-d4935173ad0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246596    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246623    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246632    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f7a4da-4b35-47e5-b761-831212f95f0b-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246640    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxn5\" (UniqueName: \"kubernetes.io/projected/76f7a4da-4b35-47e5-b761-831212f95f0b-kube-api-access-fdxn5\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246651    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f7a4da-4b35-47e5-b761-831212f95f0b-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246661    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7rlw\" (UniqueName: \"kubernetes.io/projected/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-kube-api-access-x7rlw\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246668    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.246682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.246678    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:23.611300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.611202    2551 generic.go:356] "Generic (PLEG): container finished" podID="76f7a4da-4b35-47e5-b761-831212f95f0b" containerID="bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c" exitCode=0
Sep 15 21:38:23.611300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.611252    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"
Sep 15 21:38:23.611300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.611288    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6" event={"ID":"76f7a4da-4b35-47e5-b761-831212f95f0b","Type":"ContainerDied","Data":"bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c"}
Sep 15 21:38:23.611566 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.611337    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6" event={"ID":"76f7a4da-4b35-47e5-b761-831212f95f0b","Type":"ContainerDied","Data":"aff7e442dd089946578f27526cdc10e47529c60723448916784f805d1afebf6d"}
Sep 15 21:38:23.611566 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.611362    2551 scope.go:117] "RemoveContainer" containerID="bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c"
Sep 15 21:38:23.612880 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.612858    2551 generic.go:356] "Generic (PLEG): container finished" podID="ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" containerID="1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115" exitCode=0
Sep 15 21:38:23.613009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.612897    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"
Sep 15 21:38:23.613009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.612910    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt" event={"ID":"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f","Type":"ContainerDied","Data":"1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115"}
Sep 15 21:38:23.613009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.612938    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt" event={"ID":"ec29a1f6-8d6a-4112-a1ff-d4935173ad0f","Type":"ContainerDied","Data":"98579f27c862af772e8a5d6f63d38794cc728191eec7c58acbe27ad90fe63eb1"}
Sep 15 21:38:23.621206 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.621175    2551 scope.go:117] "RemoveContainer" containerID="bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c"
Sep 15 21:38:23.621499 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:23.621478    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c\": container with ID starting with bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c not found: ID does not exist" containerID="bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c"
Sep 15 21:38:23.621578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.621512    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c"} err="failed to get container status \"bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c\": rpc error: code = NotFound desc = could not find container \"bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c\": container with ID starting with bfe9f4205382de7cadb34e3246a61a277952888912269a31f6690b2a428b7a1c not found: ID does not exist"
Sep 15 21:38:23.621578 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.621534    2551 scope.go:117] "RemoveContainer" containerID="1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115"
Sep 15 21:38:23.629443 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.629426    2551 scope.go:117] "RemoveContainer" containerID="1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115"
Sep 15 21:38:23.629731 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:23.629709    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115\": container with ID starting with 1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115 not found: ID does not exist" containerID="1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115"
Sep 15 21:38:23.629826 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.629737    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115"} err="failed to get container status \"1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115\": rpc error: code = NotFound desc = could not find container \"1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115\": container with ID starting with 1a3ad812e34b8ba01ebfd06336f32e35e92ab4469911b90ee566ec1ad4c50115 not found: ID does not exist"
Sep 15 21:38:23.636183 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.636160    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"]
Sep 15 21:38:23.638051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.638030    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-625609/distinct-w4-625609-kserve-6fd75874c9-gg6gt"]
Sep 15 21:38:23.647322 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.647299    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"]
Sep 15 21:38:23.651274 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.651253    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-625609/simulator-w4-625609-kserve-977dd4d4f-d4rk6"]
Sep 15 21:38:23.769764 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.769735    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f7a4da-4b35-47e5-b761-831212f95f0b" path="/var/lib/kubelet/pods/76f7a4da-4b35-47e5-b761-831212f95f0b/volumes"
Sep 15 21:38:23.770154 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:23.770140    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" path="/var/lib/kubelet/pods/ec29a1f6-8d6a-4112-a1ff-d4935173ad0f/volumes"
Sep 15 21:38:25.675087 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:25.675052    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"]
Sep 15 21:38:25.675604 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:25.675292    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" containerID="cri-o://f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3" gracePeriod=30
Sep 15 21:38:29.924323 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:29.924287    2551 patch_prober.go:28] interesting pod/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:38:29.924749 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:29.924346    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:38:30.929259 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:30.929235    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:38:31.026398 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026311    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pff5q\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-kube-api-access-pff5q\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026398 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026350    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-istio-token\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026398 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026375    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-data\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026398 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026394    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2726b1ad-32a6-400a-a695-18b14674eed5-istiod-ca-cert\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026412    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-envoy\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026436    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-certs\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026489    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-socket\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026511    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-credential-socket\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026554    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2726b1ad-32a6-400a-a695-18b14674eed5-istio-podinfo\") pod \"2726b1ad-32a6-400a-a695-18b14674eed5\" (UID: \"2726b1ad-32a6-400a-a695-18b14674eed5\") "
Sep 15 21:38:31.026978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026832    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:31.026978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026907    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:31.026978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026944    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:31.026978 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.026970    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-data" (OuterVolumeSpecName: "istio-data") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:31.027383 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.027271    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2726b1ad-32a6-400a-a695-18b14674eed5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:31.028713 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.028690    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-istio-token" (OuterVolumeSpecName: "istio-token") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:31.028839 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.028753    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-kube-api-access-pff5q" (OuterVolumeSpecName: "kube-api-access-pff5q") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "kube-api-access-pff5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:31.028839 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.028827    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:31.029009 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.028987    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2726b1ad-32a6-400a-a695-18b14674eed5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2726b1ad-32a6-400a-a695-18b14674eed5" (UID: "2726b1ad-32a6-400a-a695-18b14674eed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:31.128327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128290    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128320    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128330    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2726b1ad-32a6-400a-a695-18b14674eed5-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128341    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pff5q\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-kube-api-access-pff5q\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128350    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2726b1ad-32a6-400a-a695-18b14674eed5-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128359    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128367    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2726b1ad-32a6-400a-a695-18b14674eed5-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128374    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.128548 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.128381    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2726b1ad-32a6-400a-a695-18b14674eed5-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:31.656961 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.656925    2551 generic.go:356] "Generic (PLEG): container finished" podID="2726b1ad-32a6-400a-a695-18b14674eed5" containerID="f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3" exitCode=0
Sep 15 21:38:31.656961 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.656966    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"
Sep 15 21:38:31.657254 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.656973    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" event={"ID":"2726b1ad-32a6-400a-a695-18b14674eed5","Type":"ContainerDied","Data":"f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3"}
Sep 15 21:38:31.657254 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.657026    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f" event={"ID":"2726b1ad-32a6-400a-a695-18b14674eed5","Type":"ContainerDied","Data":"63444e94080f7a114f7f917c1bc11f0349efc48656723a97a8012207757eeffa"}
Sep 15 21:38:31.657254 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.657055    2551 scope.go:117] "RemoveContainer" containerID="f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3"
Sep 15 21:38:31.666492 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.666473    2551 scope.go:117] "RemoveContainer" containerID="7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"
Sep 15 21:38:31.676751 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.676726    2551 scope.go:117] "RemoveContainer" containerID="f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3"
Sep 15 21:38:31.677329 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:31.677129    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3\": container with ID starting with f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3 not found: ID does not exist" containerID="f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3"
Sep 15 21:38:31.677329 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.677184    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3"} err="failed to get container status \"f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3\": rpc error: code = NotFound desc = could not find container \"f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3\": container with ID starting with f5e34707a68aab0b4124657dd830dd96048e6dc42c919dbc6cacdca57acf86b3 not found: ID does not exist"
Sep 15 21:38:31.677329 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.677234    2551 scope.go:117] "RemoveContainer" containerID="7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"
Sep 15 21:38:31.678003 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:31.677961    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b\": container with ID starting with 7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b not found: ID does not exist" containerID="7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"
Sep 15 21:38:31.678106 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.678000    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b"} err="failed to get container status \"7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b\": rpc error: code = NotFound desc = could not find container \"7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b\": container with ID starting with 7c2ccac4dce0be3f8e747f55d78d0dc709294a904a10a33f28f6fd7445eba81b not found: ID does not exist"
Sep 15 21:38:31.679899 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.679875    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"]
Sep 15 21:38:31.686120 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.686100    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-625609-openshift-default-58498649b8-d7g8f"]
Sep 15 21:38:31.768049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:31.768006    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" path="/var/lib/kubelet/pods/2726b1ad-32a6-400a-a695-18b14674eed5/volumes"
Sep 15 21:38:34.026156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:34.026116    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"]
Sep 15 21:38:34.026740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:34.026390    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" containerID="cri-o://17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f" gracePeriod=30
Sep 15 21:38:37.442825 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:37.442785    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:38:37.443266 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:37.442850    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:38:39.275397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.275365    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:38:39.405556 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405474    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-podinfo\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405556 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405541    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x75qg\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-kube-api-access-x75qg\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405563    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istiod-ca-cert\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405584    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-data\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405603    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-credential-socket\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405678    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-certs\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405713    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-token\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405749    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-socket\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.405804 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.405772    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-envoy\") pod \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\" (UID: \"1b7bdbac-7830-4314-a910-ddcd3bb9c73f\") "
Sep 15 21:38:39.406724 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.406033    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:39.406724 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.406502    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:39.406724 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.406674    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:39.407300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.406844    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:39.407300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.406007    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:39.414958 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.414899    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:39.414958 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.414934    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:39.414958 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.414945    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-kube-api-access-x75qg" (OuterVolumeSpecName: "kube-api-access-x75qg") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "kube-api-access-x75qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:39.415229 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.414966    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b7bdbac-7830-4314-a910-ddcd3bb9c73f" (UID: "1b7bdbac-7830-4314-a910-ddcd3bb9c73f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:39.508668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508633    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508662    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75qg\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-kube-api-access-x75qg\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508673    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508681    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508690    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508698    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508706    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508713    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.508896 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.508722    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7bdbac-7830-4314-a910-ddcd3bb9c73f-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:39.698294 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.698253    2551 generic.go:356] "Generic (PLEG): container finished" podID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerID="17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f" exitCode=0
Sep 15 21:38:39.698474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.698324    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"
Sep 15 21:38:39.698474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.698336    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" event={"ID":"1b7bdbac-7830-4314-a910-ddcd3bb9c73f","Type":"ContainerDied","Data":"17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f"}
Sep 15 21:38:39.698474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.698421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x" event={"ID":"1b7bdbac-7830-4314-a910-ddcd3bb9c73f","Type":"ContainerDied","Data":"f5b5e986bc153e3dcd63f933f20f3895a9f7619a9a1873705a74f1fe2eb3ccd4"}
Sep 15 21:38:39.698474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.698445    2551 scope.go:117] "RemoveContainer" containerID="17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f"
Sep 15 21:38:39.708060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.708039    2551 scope.go:117] "RemoveContainer" containerID="e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"
Sep 15 21:38:39.716743 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.716718    2551 scope.go:117] "RemoveContainer" containerID="17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f"
Sep 15 21:38:39.717071 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:39.717053    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f\": container with ID starting with 17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f not found: ID does not exist" containerID="17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f"
Sep 15 21:38:39.717139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.717079    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f"} err="failed to get container status \"17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f\": rpc error: code = NotFound desc = could not find container \"17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f\": container with ID starting with 17340bad7966d028b775ec98152ebc3387847b3bd5f5e49a1e1bc9502b6fbd8f not found: ID does not exist"
Sep 15 21:38:39.717139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.717096    2551 scope.go:117] "RemoveContainer" containerID="e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"
Sep 15 21:38:39.717513 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:39.717481    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0\": container with ID starting with e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0 not found: ID does not exist" containerID="e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"
Sep 15 21:38:39.717622 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.717524    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0"} err="failed to get container status \"e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0\": rpc error: code = NotFound desc = could not find container \"e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0\": container with ID starting with e5f6810a2f17a7a504a2c253556abfd397c600294a6baeed9b2f173d51d8a4e0 not found: ID does not exist"
Sep 15 21:38:39.725460 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.725431    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"]
Sep 15 21:38:39.729827 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.729804    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9aeead-openshift-default-5bf8cdffb6-5nx4x"]
Sep 15 21:38:39.769408 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:39.769379    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" path="/var/lib/kubelet/pods/1b7bdbac-7830-4314-a910-ddcd3bb9c73f/volumes"
Sep 15 21:38:41.137615 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:41.137581    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"]
Sep 15 21:38:56.157699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.157636    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l" podUID="e49dd3a5-933f-42ab-a863-c45c3af0a7f9" containerName="main" containerID="cri-o://5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2" gracePeriod=45
Sep 15 21:38:56.417830 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.417765    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:38:56.576132 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576089    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tmp-dir\") pod \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") "
Sep 15 21:38:56.576355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576136    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-home\") pod \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") "
Sep 15 21:38:56.576355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576248    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-dshm\") pod \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") "
Sep 15 21:38:56.576355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576288    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-model-cache\") pod \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") "
Sep 15 21:38:56.576355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576336    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tls-certs\") pod \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") "
Sep 15 21:38:56.576561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576403    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbmh4\" (UniqueName: \"kubernetes.io/projected/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-kube-api-access-jbmh4\") pod \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\" (UID: \"e49dd3a5-933f-42ab-a863-c45c3af0a7f9\") "
Sep 15 21:38:56.576561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576410    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-home" (OuterVolumeSpecName: "home") pod "e49dd3a5-933f-42ab-a863-c45c3af0a7f9" (UID: "e49dd3a5-933f-42ab-a863-c45c3af0a7f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.576561 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576426    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e49dd3a5-933f-42ab-a863-c45c3af0a7f9" (UID: "e49dd3a5-933f-42ab-a863-c45c3af0a7f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.576716 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576579    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e49dd3a5-933f-42ab-a863-c45c3af0a7f9" (UID: "e49dd3a5-933f-42ab-a863-c45c3af0a7f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.576771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576729    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.576771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576747    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.576771 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.576759    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.578599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.578574    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-dshm" (OuterVolumeSpecName: "dshm") pod "e49dd3a5-933f-42ab-a863-c45c3af0a7f9" (UID: "e49dd3a5-933f-42ab-a863-c45c3af0a7f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:56.578705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.578630    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-kube-api-access-jbmh4" (OuterVolumeSpecName: "kube-api-access-jbmh4") pod "e49dd3a5-933f-42ab-a863-c45c3af0a7f9" (UID: "e49dd3a5-933f-42ab-a863-c45c3af0a7f9"). InnerVolumeSpecName "kube-api-access-jbmh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:56.578705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.578640    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e49dd3a5-933f-42ab-a863-c45c3af0a7f9" (UID: "e49dd3a5-933f-42ab-a863-c45c3af0a7f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:38:56.678018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.677935    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmh4\" (UniqueName: \"kubernetes.io/projected/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-kube-api-access-jbmh4\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.678018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.677963    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.678018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.677974    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49dd3a5-933f-42ab-a863-c45c3af0a7f9-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:38:56.783865 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.783832    2551 generic.go:356] "Generic (PLEG): container finished" podID="e49dd3a5-933f-42ab-a863-c45c3af0a7f9" containerID="5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2" exitCode=0
Sep 15 21:38:56.784019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.783871    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"
Sep 15 21:38:56.784019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.783908    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l" event={"ID":"e49dd3a5-933f-42ab-a863-c45c3af0a7f9","Type":"ContainerDied","Data":"5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2"}
Sep 15 21:38:56.784019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.783948    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l" event={"ID":"e49dd3a5-933f-42ab-a863-c45c3af0a7f9","Type":"ContainerDied","Data":"78756dcda4723a44d75ff1d9a98690e57a3e3a1366c0cb332dee9e4744eaff0a"}
Sep 15 21:38:56.784019 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.783965    2551 scope.go:117] "RemoveContainer" containerID="5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2"
Sep 15 21:38:56.793118 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.793101    2551 scope.go:117] "RemoveContainer" containerID="5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2"
Sep 15 21:38:56.793435 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:38:56.793413    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2\": container with ID starting with 5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2 not found: ID does not exist" containerID="5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2"
Sep 15 21:38:56.793501 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.793445    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2"} err="failed to get container status \"5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2\": rpc error: code = NotFound desc = could not find container \"5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2\": container with ID starting with 5fcc9bc1a1db4e53081ed8b629accecc208ebc7afaf9c878b0777befd6b7e5d2 not found: ID does not exist"
Sep 15 21:38:56.804797 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.804764    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"]
Sep 15 21:38:56.808774 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:56.808751    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c4dc8/auth-test-model-3c4dc8-kserve-98b4d6fc9-lmx2l"]
Sep 15 21:38:57.768893 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:38:57.768856    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49dd3a5-933f-42ab-a863-c45c3af0a7f9" path="/var/lib/kubelet/pods/e49dd3a5-933f-42ab-a863-c45c3af0a7f9/volumes"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238103    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"]
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238630    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238650    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238664    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238672    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238682    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238691    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238704    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238712    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238724    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94712580-5511-48e1-9313-6082aa13b707" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238732    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="94712580-5511-48e1-9313-6082aa13b707" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238741    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238749    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238761    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238769    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238777    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f7a4da-4b35-47e5-b761-831212f95f0b" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238784    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f7a4da-4b35-47e5-b761-831212f95f0b" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238808    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238816    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238826    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238835    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238844    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238852    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238860    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238868    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238877    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882a2d76-cecd-49d9-b092-b777907ce81f" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238885    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="882a2d76-cecd-49d9-b092-b777907ce81f" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238897    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238904    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238917    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238925    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238939    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49dd3a5-933f-42ab-a863-c45c3af0a7f9" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.238947    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49dd3a5-933f-42ab-a863-c45c3af0a7f9" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239034    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239047    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239056    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49dd3a5-933f-42ab-a863-c45c3af0a7f9" containerName="main"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239066    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy"
Sep 15 21:39:18.239351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239074    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f77b46f-ea22-43f8-8105-c25ea9cabf4a" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239080    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239088    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f7a4da-4b35-47e5-b761-831212f95f0b" containerName="main"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239096    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec29a1f6-8d6a-4112-a1ff-d4935173ad0f" containerName="main"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239104    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239112    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="882a2d76-cecd-49d9-b092-b777907ce81f" containerName="main"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239120    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239127    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="94712580-5511-48e1-9313-6082aa13b707" containerName="main"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239135    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac62d37-8f52-4ccf-acae-5e56bf196b4c" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239145    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7bdbac-7830-4314-a910-ddcd3bb9c73f" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239153    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d547ba7-f53f-4638-9960-746741734ddf" containerName="istio-proxy"
Sep 15 21:39:18.241859 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.239161    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy"
Sep 15 21:39:18.243694 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.243673    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.247159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.246923    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7c57b6\"/\"default-dockercfg-rvhzm\""
Sep 15 21:39:18.247159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.247021    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7c57b6\"/\"openshift-service-ca.crt\""
Sep 15 21:39:18.247159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.247028    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7c57b6\"/\"kube-root-ca.crt\""
Sep 15 21:39:18.247494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.247427    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7c57b6\"/\"sub-test-model-7c57b6-kserve-self-signed-certs\""
Sep 15 21:39:18.250139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.250113    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"]
Sep 15 21:39:18.370308 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.370266    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khx4f\" (UniqueName: \"kubernetes.io/projected/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-kube-api-access-khx4f\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.370494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.370328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-model-cache\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.370494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.370369    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-home\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.370494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.370438    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tls-certs\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.370623 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.370529    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-dshm\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.370623 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.370569    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tmp-dir\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.471549 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.471509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tls-certs\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.471755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.471579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-dshm\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.471755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.471618    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tmp-dir\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.471755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.471668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khx4f\" (UniqueName: \"kubernetes.io/projected/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-kube-api-access-khx4f\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.471755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.471705    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-model-cache\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.471755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.471740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-home\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.472102 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.472077    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tmp-dir\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.472183 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.472111    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-home\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.472183 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.472168    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-model-cache\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.474122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.474099    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-dshm\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.474348 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.474331    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tls-certs\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.480183 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.480162    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khx4f\" (UniqueName: \"kubernetes.io/projected/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-kube-api-access-khx4f\") pod \"sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") " pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.555158 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.555070    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:18.833407 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.833383    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"]
Sep 15 21:39:18.835825 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:39:18.835796    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6961f11e_8ad1_44d6_b49e_2ff7b93fdcca.slice/crio-cd5615433af6685a07e3f954987969973f8501fcd70d67bc9e9622760c05eb56 WatchSource:0}: Error finding container cd5615433af6685a07e3f954987969973f8501fcd70d67bc9e9622760c05eb56: Status 404 returned error can't find the container with id cd5615433af6685a07e3f954987969973f8501fcd70d67bc9e9622760c05eb56
Sep 15 21:39:18.897062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:18.897035    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd" event={"ID":"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca","Type":"ContainerStarted","Data":"cd5615433af6685a07e3f954987969973f8501fcd70d67bc9e9622760c05eb56"}
Sep 15 21:39:19.903857 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:19.903814    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd" event={"ID":"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca","Type":"ContainerStarted","Data":"99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab"}
Sep 15 21:39:19.921207 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:19.921135    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd" podStartSLOduration=1.921117899 podStartE2EDuration="1.921117899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:19.92028385 +0000 UTC m=+1858.722152795" watchObservedRunningTime="2026-09-15 21:39:19.921117899 +0000 UTC m=+1858.722986842"
Sep 15 21:39:28.556237 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:28.556169    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:28.556237 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:28.556241    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:28.569931 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:28.569899    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:28.960325 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:28.960297    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:39:55.999310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:55.999229    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7448-96ba45"]
Sep 15 21:39:55.999688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:55.999573    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy"
Sep 15 21:39:55.999688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:55.999586    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9680b161-7c0c-4fb1-8784-66a6a0ccd9d1" containerName="istio-proxy"
Sep 15 21:39:55.999688 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:55.999641    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2726b1ad-32a6-400a-a695-18b14674eed5" containerName="istio-proxy"
Sep 15 21:39:56.002496 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.002480    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:39:56.004834 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.004813    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mbx7k\""
Sep 15 21:39:56.008450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.008430    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7448-96ba45"]
Sep 15 21:39:56.021567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.021543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6jg\" (UniqueName: \"kubernetes.io/projected/36bbbb19-add3-4c4d-b226-910c82808e80-kube-api-access-hw6jg\") pod \"test-curl-7448-96ba45\" (UID: \"36bbbb19-add3-4c4d-b226-910c82808e80\") " pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:39:56.122556 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.122523    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6jg\" (UniqueName: \"kubernetes.io/projected/36bbbb19-add3-4c4d-b226-910c82808e80-kube-api-access-hw6jg\") pod \"test-curl-7448-96ba45\" (UID: \"36bbbb19-add3-4c4d-b226-910c82808e80\") " pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:39:56.130434 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.130397    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6jg\" (UniqueName: \"kubernetes.io/projected/36bbbb19-add3-4c4d-b226-910c82808e80-kube-api-access-hw6jg\") pod \"test-curl-7448-96ba45\" (UID: \"36bbbb19-add3-4c4d-b226-910c82808e80\") " pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:39:56.312841 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.312760    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:39:56.580932 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:56.580785    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7448-96ba45"]
Sep 15 21:39:56.583283 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:39:56.583251    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36bbbb19_add3_4c4d_b226_910c82808e80.slice/crio-c90d46b1a4b712702f4a8be9d5cc8edc300de4f6bc652ac951673b557afc309a WatchSource:0}: Error finding container c90d46b1a4b712702f4a8be9d5cc8edc300de4f6bc652ac951673b557afc309a: Status 404 returned error can't find the container with id c90d46b1a4b712702f4a8be9d5cc8edc300de4f6bc652ac951673b557afc309a
Sep 15 21:39:57.088619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.088584    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7448-96ba45" event={"ID":"36bbbb19-add3-4c4d-b226-910c82808e80","Type":"ContainerStarted","Data":"3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f"}
Sep 15 21:39:57.088619 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.088627    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7448-96ba45" event={"ID":"36bbbb19-add3-4c4d-b226-910c82808e80","Type":"ContainerStarted","Data":"c90d46b1a4b712702f4a8be9d5cc8edc300de4f6bc652ac951673b557afc309a"}
Sep 15 21:39:57.104352 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.104300    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7448-96ba45" podStartSLOduration=2.104283781 podStartE2EDuration="2.104283781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:57.103085897 +0000 UTC m=+1895.904954841" watchObservedRunningTime="2026-09-15 21:39:57.104283781 +0000 UTC m=+1895.906152754"
Sep 15 21:39:57.704552 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.704521    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7448-96ba45"]
Sep 15 21:39:57.708262 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.708236    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7448-96ba45"]
Sep 15 21:39:57.825350 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.825271    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7448-cc8416"]
Sep 15 21:39:57.825613 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.825601    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36bbbb19-add3-4c4d-b226-910c82808e80" containerName="test-curl-7448-96ba45"
Sep 15 21:39:57.825665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.825615    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="36bbbb19-add3-4c4d-b226-910c82808e80" containerName="test-curl-7448-96ba45"
Sep 15 21:39:57.825700 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.825676    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="36bbbb19-add3-4c4d-b226-910c82808e80" containerName="test-curl-7448-96ba45"
Sep 15 21:39:57.828599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.828582    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:39:57.834986 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.834962    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7448-cc8416"]
Sep 15 21:39:57.940156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:57.940117    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65h5j\" (UniqueName: \"kubernetes.io/projected/0b64c99b-b392-450b-9f72-02fd68addaee-kube-api-access-65h5j\") pod \"test-curl-7448-cc8416\" (UID: \"0b64c99b-b392-450b-9f72-02fd68addaee\") " pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:39:58.041135 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:58.041101    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65h5j\" (UniqueName: \"kubernetes.io/projected/0b64c99b-b392-450b-9f72-02fd68addaee-kube-api-access-65h5j\") pod \"test-curl-7448-cc8416\" (UID: \"0b64c99b-b392-450b-9f72-02fd68addaee\") " pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:39:58.057419 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:58.057388    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65h5j\" (UniqueName: \"kubernetes.io/projected/0b64c99b-b392-450b-9f72-02fd68addaee-kube-api-access-65h5j\") pod \"test-curl-7448-cc8416\" (UID: \"0b64c99b-b392-450b-9f72-02fd68addaee\") " pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:39:58.093913 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:39:58.093840    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7448-96ba45" container="test-curl-7448-96ba45"
Sep 15 21:39:58.139972 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:58.139942    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:39:58.412857 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:58.412830    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7448-cc8416"]
Sep 15 21:39:58.414891 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:39:58.414860    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b64c99b_b392_450b_9f72_02fd68addaee.slice/crio-f86b0302db26ba64400232b15de127712d9ebc9cb7b3e4aaf9118e6d8c03bdb1 WatchSource:0}: Error finding container f86b0302db26ba64400232b15de127712d9ebc9cb7b3e4aaf9118e6d8c03bdb1: Status 404 returned error can't find the container with id f86b0302db26ba64400232b15de127712d9ebc9cb7b3e4aaf9118e6d8c03bdb1
Sep 15 21:39:59.100104 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:59.100069    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7448-cc8416" event={"ID":"0b64c99b-b392-450b-9f72-02fd68addaee","Type":"ContainerStarted","Data":"9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f"}
Sep 15 21:39:59.100559 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:59.100112    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7448-cc8416" event={"ID":"0b64c99b-b392-450b-9f72-02fd68addaee","Type":"ContainerStarted","Data":"f86b0302db26ba64400232b15de127712d9ebc9cb7b3e4aaf9118e6d8c03bdb1"}
Sep 15 21:39:59.100559 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:59.100147    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7448-96ba45" podUID="36bbbb19-add3-4c4d-b226-910c82808e80" containerName="test-curl-7448-96ba45" containerID="cri-o://3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f" gracePeriod=2
Sep 15 21:39:59.114644 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:59.114596    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7448-cc8416" podStartSLOduration=2.114580356 podStartE2EDuration="2.114580356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:59.112782189 +0000 UTC m=+1897.914651131" watchObservedRunningTime="2026-09-15 21:39:59.114580356 +0000 UTC m=+1897.916449299"
Sep 15 21:39:59.665474 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:59.665436    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7448-cc8416"]
Sep 15 21:39:59.669450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:39:59.669402    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7448-cc8416"]
Sep 15 21:40:00.105314 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:40:00.105285    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7448-cc8416" container="test-curl-7448-cc8416"
Sep 15 21:40:01.110857 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:01.110821    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7448-cc8416" podUID="0b64c99b-b392-450b-9f72-02fd68addaee" containerName="test-curl-7448-cc8416" containerID="cri-o://9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f" gracePeriod=2
Sep 15 21:40:01.287599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:01.287576    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:40:01.371752 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:01.371654    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw6jg\" (UniqueName: \"kubernetes.io/projected/36bbbb19-add3-4c4d-b226-910c82808e80-kube-api-access-hw6jg\") pod \"36bbbb19-add3-4c4d-b226-910c82808e80\" (UID: \"36bbbb19-add3-4c4d-b226-910c82808e80\") "
Sep 15 21:40:01.373863 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:01.373841    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bbbb19-add3-4c4d-b226-910c82808e80-kube-api-access-hw6jg" (OuterVolumeSpecName: "kube-api-access-hw6jg") pod "36bbbb19-add3-4c4d-b226-910c82808e80" (UID: "36bbbb19-add3-4c4d-b226-910c82808e80"). InnerVolumeSpecName "kube-api-access-hw6jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:01.472534 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:01.472495    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw6jg\" (UniqueName: \"kubernetes.io/projected/36bbbb19-add3-4c4d-b226-910c82808e80-kube-api-access-hw6jg\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:01.770236 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:01.770174    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36bbbb19-add3-4c4d-b226-910c82808e80" path="/var/lib/kubelet/pods/36bbbb19-add3-4c4d-b226-910c82808e80/volumes"
Sep 15 21:40:02.116344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:02.116259    2551 generic.go:356] "Generic (PLEG): container finished" podID="36bbbb19-add3-4c4d-b226-910c82808e80" containerID="3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f" exitCode=137
Sep 15 21:40:02.116344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:02.116306    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-96ba45"
Sep 15 21:40:02.116344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:02.116331    2551 scope.go:117] "RemoveContainer" containerID="3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f"
Sep 15 21:40:02.124984 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:02.124963    2551 scope.go:117] "RemoveContainer" containerID="3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f"
Sep 15 21:40:02.125298 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:40:02.125281    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f\": container with ID starting with 3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f not found: ID does not exist" containerID="3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f"
Sep 15 21:40:02.125365 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:02.125307    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f"} err="failed to get container status \"3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f\": rpc error: code = NotFound desc = could not find container \"3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f\": container with ID starting with 3ba23e993581fe5b40badd389c73034bb2ff2e0d1fe8f1418f0638bd7aeaa50f not found: ID does not exist"
Sep 15 21:40:02.648397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:02.648354    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"]
Sep 15 21:40:03.302857 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:03.302829    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:40:03.387540 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:03.387457    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65h5j\" (UniqueName: \"kubernetes.io/projected/0b64c99b-b392-450b-9f72-02fd68addaee-kube-api-access-65h5j\") pod \"0b64c99b-b392-450b-9f72-02fd68addaee\" (UID: \"0b64c99b-b392-450b-9f72-02fd68addaee\") "
Sep 15 21:40:03.389638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:03.389614    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b64c99b-b392-450b-9f72-02fd68addaee-kube-api-access-65h5j" (OuterVolumeSpecName: "kube-api-access-65h5j") pod "0b64c99b-b392-450b-9f72-02fd68addaee" (UID: "0b64c99b-b392-450b-9f72-02fd68addaee"). InnerVolumeSpecName "kube-api-access-65h5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:03.488430 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:03.488396    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65h5j\" (UniqueName: \"kubernetes.io/projected/0b64c99b-b392-450b-9f72-02fd68addaee-kube-api-access-65h5j\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:03.768782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:03.768746    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b64c99b-b392-450b-9f72-02fd68addaee" path="/var/lib/kubelet/pods/0b64c99b-b392-450b-9f72-02fd68addaee/volumes"
Sep 15 21:40:04.127667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:04.127581    2551 generic.go:356] "Generic (PLEG): container finished" podID="0b64c99b-b392-450b-9f72-02fd68addaee" containerID="9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f" exitCode=137
Sep 15 21:40:04.127667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:04.127632    2551 scope.go:117] "RemoveContainer" containerID="9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f"
Sep 15 21:40:04.127911 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:04.127629    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-cc8416"
Sep 15 21:40:04.136220 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:04.136201    2551 scope.go:117] "RemoveContainer" containerID="9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f"
Sep 15 21:40:04.136502 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:40:04.136484    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f\": container with ID starting with 9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f not found: ID does not exist" containerID="9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f"
Sep 15 21:40:04.136547 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:04.136510    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f"} err="failed to get container status \"9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f\": rpc error: code = NotFound desc = could not find container \"9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f\": container with ID starting with 9cd99364aa47202c8e5d888ea0971ab9734deaa67c185da52da0502e39e8e46f not found: ID does not exist"
Sep 15 21:40:17.671963 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:17.671901    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd" podUID="6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" containerName="main" containerID="cri-o://99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab" gracePeriod=45
Sep 15 21:40:17.922484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:17.922428    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:40:18.029980 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.029943    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khx4f\" (UniqueName: \"kubernetes.io/projected/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-kube-api-access-khx4f\") pod \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") "
Sep 15 21:40:18.029980 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.029984    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tmp-dir\") pod \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") "
Sep 15 21:40:18.030228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030005    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-model-cache\") pod \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") "
Sep 15 21:40:18.030228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030025    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-home\") pod \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") "
Sep 15 21:40:18.030228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030116    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tls-certs\") pod \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") "
Sep 15 21:40:18.030228 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030151    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-dshm\") pod \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\" (UID: \"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca\") "
Sep 15 21:40:18.030413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030253    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" (UID: "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:18.030413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030270    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-model-cache" (OuterVolumeSpecName: "model-cache") pod "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" (UID: "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:18.030413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030402    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-home" (OuterVolumeSpecName: "home") pod "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" (UID: "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:18.030568 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030525    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:18.030568 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030544    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:18.030568 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.030553    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:18.032581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.032551    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-dshm" (OuterVolumeSpecName: "dshm") pod "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" (UID: "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:18.032581 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.032571    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" (UID: "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:40:18.032734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.032575    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-kube-api-access-khx4f" (OuterVolumeSpecName: "kube-api-access-khx4f") pod "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" (UID: "6961f11e-8ad1-44d6-b49e-2ff7b93fdcca"). InnerVolumeSpecName "kube-api-access-khx4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:18.131318 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.131277    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khx4f\" (UniqueName: \"kubernetes.io/projected/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-kube-api-access-khx4f\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:18.131318 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.131312    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:18.131318 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.131325    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:40:18.193711 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.193618    2551 generic.go:356] "Generic (PLEG): container finished" podID="6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" containerID="99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab" exitCode=0
Sep 15 21:40:18.193885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.193713    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd" event={"ID":"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca","Type":"ContainerDied","Data":"99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab"}
Sep 15 21:40:18.193885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.193728    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"
Sep 15 21:40:18.193885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.193744    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd" event={"ID":"6961f11e-8ad1-44d6-b49e-2ff7b93fdcca","Type":"ContainerDied","Data":"cd5615433af6685a07e3f954987969973f8501fcd70d67bc9e9622760c05eb56"}
Sep 15 21:40:18.193885 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.193763    2551 scope.go:117] "RemoveContainer" containerID="99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab"
Sep 15 21:40:18.202682 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.202662    2551 scope.go:117] "RemoveContainer" containerID="99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab"
Sep 15 21:40:18.202953 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:40:18.202934    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab\": container with ID starting with 99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab not found: ID does not exist" containerID="99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab"
Sep 15 21:40:18.203063 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.202967    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab"} err="failed to get container status \"99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab\": rpc error: code = NotFound desc = could not find container \"99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab\": container with ID starting with 99ade12495d0de3f73eaef2cf3ea470499fd40e622a24d13fef155bad337dfab not found: ID does not exist"
Sep 15 21:40:18.216851 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.216825    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"]
Sep 15 21:40:18.222382 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:18.222360    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7c57b6/sub-test-model-7c57b6-kserve-68c5d6dcf-g5twd"]
Sep 15 21:40:19.769412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:40:19.769380    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" path="/var/lib/kubelet/pods/6961f11e-8ad1-44d6-b49e-2ff7b93fdcca/volumes"
Sep 15 21:41:38.803725 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.803685    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"]
Sep 15 21:41:38.804273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.804181    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b64c99b-b392-450b-9f72-02fd68addaee" containerName="test-curl-7448-cc8416"
Sep 15 21:41:38.804273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.804227    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b64c99b-b392-450b-9f72-02fd68addaee" containerName="test-curl-7448-cc8416"
Sep 15 21:41:38.804273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.804238    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" containerName="main"
Sep 15 21:41:38.804273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.804244    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" containerName="main"
Sep 15 21:41:38.804436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.804304    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6961f11e-8ad1-44d6-b49e-2ff7b93fdcca" containerName="main"
Sep 15 21:41:38.804436 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.804311    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b64c99b-b392-450b-9f72-02fd68addaee" containerName="test-curl-7448-cc8416"
Sep 15 21:41:38.807317 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.807298    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.809714 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.809687    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7c0f45-openshift-default-dockercfg-zhcr8\""
Sep 15 21:41:38.818506 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.818484    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"]
Sep 15 21:41:38.861915 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.861877    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-podinfo\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862102 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.861923    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-socket\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862102 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.861945    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-credential-socket\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862102 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.861971    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-data\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862102 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.862022    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-certs\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862102 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.862072    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfht7\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-kube-api-access-sfht7\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.862118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bd7bff-d9e4-43cf-92e9-baa95871105e-istiod-ca-cert\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.862180    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-token\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.862393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.862292    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-envoy\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963065    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-envoy\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963152    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-podinfo\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963182    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-socket\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-credential-socket\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-data\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963296    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-certs\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963333 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963323    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfht7\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-kube-api-access-sfht7\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963388    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bd7bff-d9e4-43cf-92e9-baa95871105e-istiod-ca-cert\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-token\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963656    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-credential-socket\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963712    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-certs\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963850    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-data\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.963874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.963851    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-socket\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.964569 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.964546    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bd7bff-d9e4-43cf-92e9-baa95871105e-istiod-ca-cert\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.965653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.965632    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-envoy\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.965781 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.965764    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-podinfo\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.971478 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.971450    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-token\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:38.971740 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:38.971719    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfht7\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-kube-api-access-sfht7\") pod \"e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") " pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:39.118865 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.118773    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:39.394973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.394931    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"]
Sep 15 21:41:39.397245 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:41:39.397212    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bd7bff_d9e4_43cf_92e9_baa95871105e.slice/crio-13f8146e9db0518797811cb6a5cdc48954b080c530f5f8fc7fea6482bd4509a1 WatchSource:0}: Error finding container 13f8146e9db0518797811cb6a5cdc48954b080c530f5f8fc7fea6482bd4509a1: Status 404 returned error can't find the container with id 13f8146e9db0518797811cb6a5cdc48954b080c530f5f8fc7fea6482bd4509a1
Sep 15 21:41:39.399665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.399629    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:41:39.399745 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.399714    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:41:39.399788 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.399756    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:41:39.593467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.593425    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" event={"ID":"19bd7bff-d9e4-43cf-92e9-baa95871105e","Type":"ContainerStarted","Data":"c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"}
Sep 15 21:41:39.593641 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.593492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" event={"ID":"19bd7bff-d9e4-43cf-92e9-baa95871105e","Type":"ContainerStarted","Data":"13f8146e9db0518797811cb6a5cdc48954b080c530f5f8fc7fea6482bd4509a1"}
Sep 15 21:41:39.615943 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:39.615883    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podStartSLOduration=1.615861993 podStartE2EDuration="1.615861993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:39.613032864 +0000 UTC m=+1998.414901830" watchObservedRunningTime="2026-09-15 21:41:39.615861993 +0000 UTC m=+1998.417730938"
Sep 15 21:41:40.119001 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:40.118962    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:40.123843 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:40.123809    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:40.597894 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:40.597859    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:40.598777 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:40.598757    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:41:43.395356 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.395322    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"]
Sep 15 21:41:43.398609 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.398587    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.400857 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.400835    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7c0f45-serving-cert\""
Sep 15 21:41:43.409490 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.409467    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"]
Sep 15 21:41:43.511312 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.511278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af86853-ae65-46bb-9155-2cee2a4ce17e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.511527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.511375    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmkb5\" (UniqueName: \"kubernetes.io/projected/7af86853-ae65-46bb-9155-2cee2a4ce17e-kube-api-access-rmkb5\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.511527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.511407    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-metrics-certs\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.511527 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.511499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.612562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.612529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.612773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.612583    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af86853-ae65-46bb-9155-2cee2a4ce17e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.612773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.612634    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmkb5\" (UniqueName: \"kubernetes.io/projected/7af86853-ae65-46bb-9155-2cee2a4ce17e-kube-api-access-rmkb5\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.612773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.612651    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-metrics-certs\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.613421 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.613343    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af86853-ae65-46bb-9155-2cee2a4ce17e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.615266 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.615245    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-metrics-certs\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.615356 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.615284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.619863 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.619844    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmkb5\" (UniqueName: \"kubernetes.io/projected/7af86853-ae65-46bb-9155-2cee2a4ce17e-kube-api-access-rmkb5\") pod \"maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.708958 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.708926    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:43.986029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.985869    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"]
Sep 15 21:41:43.989913 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:41:43.989878    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af86853_ae65_46bb_9155_2cee2a4ce17e.slice/crio-9e571dab14e5427172d02f1490e68a4a586e689176c5b5bafedb25d6de5a988b WatchSource:0}: Error finding container 9e571dab14e5427172d02f1490e68a4a586e689176c5b5bafedb25d6de5a988b: Status 404 returned error can't find the container with id 9e571dab14e5427172d02f1490e68a4a586e689176c5b5bafedb25d6de5a988b
Sep 15 21:41:43.992351 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:43.992321    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:41:44.617855 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:44.617818    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5" event={"ID":"7af86853-ae65-46bb-9155-2cee2a4ce17e","Type":"ContainerStarted","Data":"9e571dab14e5427172d02f1490e68a4a586e689176c5b5bafedb25d6de5a988b"}
Sep 15 21:41:45.624148 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:45.624104    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5" event={"ID":"7af86853-ae65-46bb-9155-2cee2a4ce17e","Type":"ContainerStarted","Data":"d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72"}
Sep 15 21:41:45.624673 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:45.624249    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:45.642396 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:45.642342    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5" podStartSLOduration=1.968142587 podStartE2EDuration="2.642326305s" totalImagesPullingTime="674.183718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:45.639936652 +0000 UTC m=+2004.441805608" watchObservedRunningTime="2026-09-15 21:41:45.642326305 +0000 UTC m=+2004.444195293"
Sep 15 21:41:51.635820 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:51.635789    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:41:59.695060 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:59.695019    2551 generic.go:356] "Generic (PLEG): container finished" podID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerID="c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac" exitCode=0
Sep 15 21:41:59.695622 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:59.695089    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" event={"ID":"19bd7bff-d9e4-43cf-92e9-baa95871105e","Type":"ContainerDied","Data":"c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"}
Sep 15 21:41:59.695622 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:59.695612    2551 scope.go:117] "RemoveContainer" containerID="c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"
Sep 15 21:41:59.696724 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:59.696696    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:41:59.696813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:59.696763    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:41:59.696813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:41:59.696794    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:00.704367 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:00.704331    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" event={"ID":"19bd7bff-d9e4-43cf-92e9-baa95871105e","Type":"ContainerStarted","Data":"a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a"}
Sep 15 21:42:01.119136 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:01.119035    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:42:01.119378 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:01.119345    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 21:42:01.119504 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:01.119406    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 21:42:02.119653 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:02.119616    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 21:42:02.120127 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:02.119672    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 21:42:03.119787 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:03.119754    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 21:42:03.120302 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:03.119808    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 21:42:04.119409 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:04.119369    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 21:42:04.119595 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:04.119435    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 21:42:05.119291 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:05.119254    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 21:42:05.119710 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:05.119315    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 21:42:06.124239 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:06.124183    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:42:06.124646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:06.124626    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:42:06.125444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:06.125425    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:42:07.507403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.507366    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"]
Sep 15 21:42:07.510467 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.510440    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.513157 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.513133    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e985b6-openshift-default-dockercfg-zzmdv\""
Sep 15 21:42:07.525964 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.525941    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"]
Sep 15 21:42:07.550789 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.550749    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-token\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.550943 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.550808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-socket\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.550943 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.550873    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-certs\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.551051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.550975    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7dl\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-kube-api-access-rg7dl\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.551051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.551008    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-envoy\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.551051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.551030    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-data\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.551233 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.551145    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istiod-ca-cert\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.551304 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.551237    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-credential-socket\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.551359 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.551305    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-podinfo\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652577 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7dl\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-kube-api-access-rg7dl\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652585    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-envoy\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652603    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-data\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652644    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istiod-ca-cert\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652674    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-credential-socket\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-podinfo\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-token\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.652761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652754    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-socket\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.653159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.652788    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-certs\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.653159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.653093    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-data\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.653284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.653160    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-credential-socket\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.653284 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.653264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-socket\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.653558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.653339    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-certs\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.653558 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.653460    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istiod-ca-cert\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.655017 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.654996    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-envoy\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.655323 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.655307    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-podinfo\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.665574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.665549    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7dl\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-kube-api-access-rg7dl\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.665983 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.665915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-token\") pod \"e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") " pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:07.821307 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:07.821225    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:08.101222 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.101180    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"]
Sep 15 21:42:08.103084 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:42:08.103054    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138592b7_88d3_4c81_be33_4ed6a8cdbd2b.slice/crio-7a16b0fcacf88e850d0f86c75857d69f1b4f0755cd3ecdbc24fc8c8e47f99bf4 WatchSource:0}: Error finding container 7a16b0fcacf88e850d0f86c75857d69f1b4f0755cd3ecdbc24fc8c8e47f99bf4: Status 404 returned error can't find the container with id 7a16b0fcacf88e850d0f86c75857d69f1b4f0755cd3ecdbc24fc8c8e47f99bf4
Sep 15 21:42:08.105441 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.105407    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:08.105511 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.105474    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:08.105511 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.105502    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:08.744128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.744085    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" event={"ID":"138592b7-88d3-4c81-be33-4ed6a8cdbd2b","Type":"ContainerStarted","Data":"b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"}
Sep 15 21:42:08.744652 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.744155    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" event={"ID":"138592b7-88d3-4c81-be33-4ed6a8cdbd2b","Type":"ContainerStarted","Data":"7a16b0fcacf88e850d0f86c75857d69f1b4f0755cd3ecdbc24fc8c8e47f99bf4"}
Sep 15 21:42:08.764591 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.764545    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podStartSLOduration=1.76452926 podStartE2EDuration="1.76452926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:08.762866615 +0000 UTC m=+2027.564735605" watchObservedRunningTime="2026-09-15 21:42:08.76452926 +0000 UTC m=+2027.566398203"
Sep 15 21:42:08.821813 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:08.821779    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:09.827792 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:09.827761    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:10.754662 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:10.754636    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:10.755782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:10.755764    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:26.796369 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:26.796329    2551 patch_prober.go:28] interesting pod/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 21:42:26.796778 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:26.796400    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:42:28.847280 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:28.847246    2551 generic.go:356] "Generic (PLEG): container finished" podID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerID="b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220" exitCode=0
Sep 15 21:42:28.847718 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:28.847331    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" event={"ID":"138592b7-88d3-4c81-be33-4ed6a8cdbd2b","Type":"ContainerDied","Data":"b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"}
Sep 15 21:42:28.847765 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:28.847754    2551 scope.go:117] "RemoveContainer" containerID="b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"
Sep 15 21:42:28.848838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:28.848810    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:28.848892 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:28.848878    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:28.848932 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:28.848910    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:42:29.854124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:29.854091    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" event={"ID":"138592b7-88d3-4c81-be33-4ed6a8cdbd2b","Type":"ContainerStarted","Data":"0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951"}
Sep 15 21:42:30.821838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:30.821792    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:30.822074 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:30.822050    2551 patch_prober.go:28] interesting pod/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 21:42:30.822126 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:30.822101    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 21:42:31.821669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:31.821632    2551 patch_prober.go:28] interesting pod/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 21:42:31.822059 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:31.821688    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 21:42:32.822559 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:32.822519    2551 patch_prober.go:28] interesting pod/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 21:42:32.823016 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:32.822578    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 21:42:33.822418 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:33.822368    2551 patch_prober.go:28] interesting pod/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 21:42:33.822680 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:33.822442    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 21:42:34.822067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:34.822031    2551 patch_prober.go:28] interesting pod/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 21:42:34.822264 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:34.822106    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 21:42:35.825980 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:35.825950    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:35.826450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:35.826435    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:35.827264 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:35.827241    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:42:44.829677 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.829591    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"]
Sep 15 21:42:44.832737 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.832715    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:44.835149 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.835124    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7c0f45\"/\"kube-root-ca.crt\""
Sep 15 21:42:44.836224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.836184    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7c0f45\"/\"openshift-service-ca.crt\""
Sep 15 21:42:44.836355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.836206    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7c0f45\"/\"ipp-route-7c0f45-kserve-self-signed-certs\""
Sep 15 21:42:44.836355 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.836210    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7c0f45\"/\"default-dockercfg-fkrwq\""
Sep 15 21:42:44.842103 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.842081    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"]
Sep 15 21:42:44.929649 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.929612    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-home\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:44.929649 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.929645    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tls-certs\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:44.929871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.929671    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/eff64f60-5db7-400f-869f-5e19ff7e1ebd-kube-api-access-mb7xx\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:44.929871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.929713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-dshm\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:44.929871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.929752    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-model-cache\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:44.929871 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:44.929785    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tmp-dir\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.031908 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.031869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-home\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.032699 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.032675    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tls-certs\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.032894 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.032876    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/eff64f60-5db7-400f-869f-5e19ff7e1ebd-kube-api-access-mb7xx\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.033028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.033015    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-dshm\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.033273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.033255    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-model-cache\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.033452 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.033437    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tmp-dir\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.033924 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.033906    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tmp-dir\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.034044 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.032616    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-home\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.039453 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.039431    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-dshm\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.039874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.039851    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-model-cache\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.040544 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.040525    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tls-certs\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.054424 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.054387    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/eff64f60-5db7-400f-869f-5e19ff7e1ebd-kube-api-access-mb7xx\") pod \"ipp-route-7c0f45-kserve-759b64d59f-n7vpb\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") " pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.143898 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.143812    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:45.421874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.421845    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"]
Sep 15 21:42:45.423720 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:42:45.423695    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff64f60_5db7_400f_869f_5e19ff7e1ebd.slice/crio-7740d712756e033fe2926b47fb6b027d34534ba5850de23dfedf567048a80634 WatchSource:0}: Error finding container 7740d712756e033fe2926b47fb6b027d34534ba5850de23dfedf567048a80634: Status 404 returned error can't find the container with id 7740d712756e033fe2926b47fb6b027d34534ba5850de23dfedf567048a80634
Sep 15 21:42:45.945067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.945030    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb" event={"ID":"eff64f60-5db7-400f-869f-5e19ff7e1ebd","Type":"ContainerStarted","Data":"52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48"}
Sep 15 21:42:45.945067 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.945073    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb" event={"ID":"eff64f60-5db7-400f-869f-5e19ff7e1ebd","Type":"ContainerStarted","Data":"7740d712756e033fe2926b47fb6b027d34534ba5850de23dfedf567048a80634"}
Sep 15 21:42:45.965273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:45.965224    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb" podStartSLOduration=1.965209455 podStartE2EDuration="1.965209455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:45.963650244 +0000 UTC m=+2064.765519202" watchObservedRunningTime="2026-09-15 21:42:45.965209455 +0000 UTC m=+2064.767078389"
Sep 15 21:42:55.144247 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:55.144216    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:55.144643 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:55.144262    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:55.157855 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:55.157826    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:42:56.005608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:42:56.005582    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:43:22.032765 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.032736    2551 scope.go:117] "RemoveContainer" containerID="c2a887faede87b81dd7de8dd25b6768c95d022e82fe6f59ae9697d6b336ab92a"
Sep 15 21:43:22.417086 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.417057    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"]
Sep 15 21:43:22.421454 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.421435    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.424708 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.424688    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-494d21-openshift-default-dockercfg-5tsp6\""
Sep 15 21:43:22.439329 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.439302    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"]
Sep 15 21:43:22.590944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.590898    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dz7\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-kube-api-access-95dz7\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591022    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-data\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591122 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591078    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667429f6-0017-472c-a27a-d5f38af01948-istio-podinfo\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591258 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591136    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-envoy\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591258 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591183    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-socket\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-certs\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591301    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667429f6-0017-472c-a27a-d5f38af01948-istiod-ca-cert\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591374 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-istio-token\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.591523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.591468    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-credential-socket\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692263    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-data\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667429f6-0017-472c-a27a-d5f38af01948-istio-podinfo\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692392 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692374    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-envoy\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-socket\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-certs\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667429f6-0017-472c-a27a-d5f38af01948-istiod-ca-cert\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692516    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-istio-token\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692647    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-credential-socket\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.692734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692695    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dz7\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-kube-api-access-95dz7\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.693061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692769    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-data\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.693061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692891    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-socket\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.693061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.692950    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-certs\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.693235 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.693110    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-credential-socket\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.693603 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.693581    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667429f6-0017-472c-a27a-d5f38af01948-istiod-ca-cert\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.694841 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.694822    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-envoy\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.695085 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.695066    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667429f6-0017-472c-a27a-d5f38af01948-istio-podinfo\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.705295 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.705262    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dz7\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-kube-api-access-95dz7\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.705295 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.705288    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-istio-token\") pod \"e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") " pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:22.732303 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:22.732264    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:23.009508 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.009482    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"]
Sep 15 21:43:23.010947 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:43:23.010920    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667429f6_0017_472c_a27a_d5f38af01948.slice/crio-cde1e735123c3bfcc25678a9b9af015ff7f58dd829546bfcf21f823e5c4e2993 WatchSource:0}: Error finding container cde1e735123c3bfcc25678a9b9af015ff7f58dd829546bfcf21f823e5c4e2993: Status 404 returned error can't find the container with id cde1e735123c3bfcc25678a9b9af015ff7f58dd829546bfcf21f823e5c4e2993
Sep 15 21:43:23.013322 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.013285    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:43:23.013427 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.013357    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:43:23.013427 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.013393    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:43:23.133650 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.133611    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" event={"ID":"667429f6-0017-472c-a27a-d5f38af01948","Type":"ContainerStarted","Data":"c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"}
Sep 15 21:43:23.134156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.133676    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" event={"ID":"667429f6-0017-472c-a27a-d5f38af01948","Type":"ContainerStarted","Data":"cde1e735123c3bfcc25678a9b9af015ff7f58dd829546bfcf21f823e5c4e2993"}
Sep 15 21:43:23.155323 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.155264    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podStartSLOduration=1.155241867 podStartE2EDuration="1.155241867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:23.153658386 +0000 UTC m=+2101.955527348" watchObservedRunningTime="2026-09-15 21:43:23.155241867 +0000 UTC m=+2101.957110811"
Sep 15 21:43:23.733006 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:23.732970    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:24.738647 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:24.738616    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:25.143316 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:25.143186    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:25.144115 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:25.144086    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:45.246819 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:45.246786    2551 generic.go:356] "Generic (PLEG): container finished" podID="667429f6-0017-472c-a27a-d5f38af01948" containerID="c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f" exitCode=0
Sep 15 21:43:45.247293 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:45.246865    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" event={"ID":"667429f6-0017-472c-a27a-d5f38af01948","Type":"ContainerDied","Data":"c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"}
Sep 15 21:43:45.247338 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:45.247310    2551 scope.go:117] "RemoveContainer" containerID="c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"
Sep 15 21:43:45.248508 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:45.248481    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:43:45.248572 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:45.248552    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:43:45.248608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:45.248585    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 15 21:43:46.255305 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:46.255265    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" event={"ID":"667429f6-0017-472c-a27a-d5f38af01948","Type":"ContainerStarted","Data":"50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb"}
Sep 15 21:43:46.732549 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:46.732514    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:46.732770 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:46.732747    2551 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:43:46.732822 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:46.732799    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 15 21:43:47.733517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:47.733475    2551 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:43:47.733912 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:47.733535    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 15 21:43:48.733654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:48.733612    2551 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:43:48.734064 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:48.733678    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 15 21:43:49.732863 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:49.732825    2551 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:43:49.733049 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:49.732883    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 15 21:43:50.733387 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:50.733350    2551 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:43:50.733825 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:50.733431    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 15 21:43:51.736751 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:51.736721    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:51.737128 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:51.737003    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:43:51.737835 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:43:51.737816    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:44:13.578459 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:13.578383    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"]
Sep 15 21:44:13.579007 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:13.578643    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy" containerID="cri-o://50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb" gracePeriod=30
Sep 15 21:44:18.826909 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.826882    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:44:18.994584 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994548    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-credential-socket\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.994773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994607    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-data\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.994773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994646    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-envoy\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.994773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994665    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-istio-token\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.994773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994730    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667429f6-0017-472c-a27a-d5f38af01948-istio-podinfo\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.994976 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994784    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-socket\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.994976 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994877    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:18.994976 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994900    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-data" (OuterVolumeSpecName: "istio-data") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:18.994976 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994931    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95dz7\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-kube-api-access-95dz7\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.995222 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.994988    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-certs\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.995222 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995016    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667429f6-0017-472c-a27a-d5f38af01948-istiod-ca-cert\") pod \"667429f6-0017-472c-a27a-d5f38af01948\" (UID: \"667429f6-0017-472c-a27a-d5f38af01948\") "
Sep 15 21:44:18.995222 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995106    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:18.995481 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995434    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:18.995481 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995456    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:18.995481 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995470    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:18.995667 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995628    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667429f6-0017-472c-a27a-d5f38af01948-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:18.995744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.995726    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:18.997169 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.997138    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-kube-api-access-95dz7" (OuterVolumeSpecName: "kube-api-access-95dz7") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "kube-api-access-95dz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:18.997334 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.997170    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:18.997582 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.997560    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-istio-token" (OuterVolumeSpecName: "istio-token") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:18.997582 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:18.997565    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/667429f6-0017-472c-a27a-d5f38af01948-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "667429f6-0017-472c-a27a-d5f38af01948" (UID: "667429f6-0017-472c-a27a-d5f38af01948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:44:19.096484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.096434    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95dz7\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-kube-api-access-95dz7\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:19.096484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.096477    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:19.096484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.096486    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667429f6-0017-472c-a27a-d5f38af01948-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:19.096484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.096494    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667429f6-0017-472c-a27a-d5f38af01948-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:19.096484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.096501    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667429f6-0017-472c-a27a-d5f38af01948-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:19.096803 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.096509    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667429f6-0017-472c-a27a-d5f38af01948-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:19.431824 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.431790    2551 generic.go:356] "Generic (PLEG): container finished" podID="667429f6-0017-472c-a27a-d5f38af01948" containerID="50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb" exitCode=0
Sep 15 21:44:19.432012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.431850    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"
Sep 15 21:44:19.432012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.431883    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" event={"ID":"667429f6-0017-472c-a27a-d5f38af01948","Type":"ContainerDied","Data":"50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb"}
Sep 15 21:44:19.432012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.431955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk" event={"ID":"667429f6-0017-472c-a27a-d5f38af01948","Type":"ContainerDied","Data":"cde1e735123c3bfcc25678a9b9af015ff7f58dd829546bfcf21f823e5c4e2993"}
Sep 15 21:44:19.432012 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.431978    2551 scope.go:117] "RemoveContainer" containerID="50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb"
Sep 15 21:44:19.441519 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.441501    2551 scope.go:117] "RemoveContainer" containerID="c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"
Sep 15 21:44:19.449287 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.449273    2551 scope.go:117] "RemoveContainer" containerID="50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb"
Sep 15 21:44:19.449540 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:44:19.449522    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb\": container with ID starting with 50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb not found: ID does not exist" containerID="50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb"
Sep 15 21:44:19.449588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.449550    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb"} err="failed to get container status \"50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb\": rpc error: code = NotFound desc = could not find container \"50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb\": container with ID starting with 50fe031f93e5a1212a291b3aca32eefda11d22e7d62d7b8515fc50b90ee11abb not found: ID does not exist"
Sep 15 21:44:19.449588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.449567    2551 scope.go:117] "RemoveContainer" containerID="c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"
Sep 15 21:44:19.449830 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:44:19.449810    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f\": container with ID starting with c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f not found: ID does not exist" containerID="c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"
Sep 15 21:44:19.449889 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.449833    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f"} err="failed to get container status \"c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f\": rpc error: code = NotFound desc = could not find container \"c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f\": container with ID starting with c430c26af229f5d95dfc8d1aa207998082cbc0c71f9839fac40b2f2d06976a0f not found: ID does not exist"
Sep 15 21:44:19.454309 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.454288    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"]
Sep 15 21:44:19.460232 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.460211    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-494d21-openshift-default-b4bdf8984-4fzgk"]
Sep 15 21:44:19.769159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:19.769063    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667429f6-0017-472c-a27a-d5f38af01948" path="/var/lib/kubelet/pods/667429f6-0017-472c-a27a-d5f38af01948/volumes"
Sep 15 21:44:32.687806 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:32.687769    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"]
Sep 15 21:44:32.688337 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:32.688063    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5" podUID="7af86853-ae65-46bb-9155-2cee2a4ce17e" containerName="maas-api" containerID="cri-o://d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72" gracePeriod=30
Sep 15 21:44:32.940654 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:32.940590    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:44:33.029100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.029065    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmkb5\" (UniqueName: \"kubernetes.io/projected/7af86853-ae65-46bb-9155-2cee2a4ce17e-kube-api-access-rmkb5\") pod \"7af86853-ae65-46bb-9155-2cee2a4ce17e\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") "
Sep 15 21:44:33.029298 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.029123    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-metrics-certs\") pod \"7af86853-ae65-46bb-9155-2cee2a4ce17e\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") "
Sep 15 21:44:33.029298 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.029216    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af86853-ae65-46bb-9155-2cee2a4ce17e-openshift-service-ca\") pod \"7af86853-ae65-46bb-9155-2cee2a4ce17e\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") "
Sep 15 21:44:33.029298 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.029289    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-maas-api-tls\") pod \"7af86853-ae65-46bb-9155-2cee2a4ce17e\" (UID: \"7af86853-ae65-46bb-9155-2cee2a4ce17e\") "
Sep 15 21:44:33.029588 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.029563    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af86853-ae65-46bb-9155-2cee2a4ce17e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7af86853-ae65-46bb-9155-2cee2a4ce17e" (UID: "7af86853-ae65-46bb-9155-2cee2a4ce17e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:33.029672 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.029658    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af86853-ae65-46bb-9155-2cee2a4ce17e-openshift-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:33.031524 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.031487    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7af86853-ae65-46bb-9155-2cee2a4ce17e" (UID: "7af86853-ae65-46bb-9155-2cee2a4ce17e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:33.031524 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.031505    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af86853-ae65-46bb-9155-2cee2a4ce17e-kube-api-access-rmkb5" (OuterVolumeSpecName: "kube-api-access-rmkb5") pod "7af86853-ae65-46bb-9155-2cee2a4ce17e" (UID: "7af86853-ae65-46bb-9155-2cee2a4ce17e"). InnerVolumeSpecName "kube-api-access-rmkb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:33.031524 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.031509    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7af86853-ae65-46bb-9155-2cee2a4ce17e" (UID: "7af86853-ae65-46bb-9155-2cee2a4ce17e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:33.131025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.130989    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmkb5\" (UniqueName: \"kubernetes.io/projected/7af86853-ae65-46bb-9155-2cee2a4ce17e-kube-api-access-rmkb5\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:33.131025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.131020    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-metrics-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:33.131025 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.131030    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7af86853-ae65-46bb-9155-2cee2a4ce17e-maas-api-tls\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:33.500315 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.500281    2551 generic.go:356] "Generic (PLEG): container finished" podID="7af86853-ae65-46bb-9155-2cee2a4ce17e" containerID="d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72" exitCode=0
Sep 15 21:44:33.500497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.500328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5" event={"ID":"7af86853-ae65-46bb-9155-2cee2a4ce17e","Type":"ContainerDied","Data":"d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72"}
Sep 15 21:44:33.500497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.500375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5" event={"ID":"7af86853-ae65-46bb-9155-2cee2a4ce17e","Type":"ContainerDied","Data":"9e571dab14e5427172d02f1490e68a4a586e689176c5b5bafedb25d6de5a988b"}
Sep 15 21:44:33.500497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.500403    2551 scope.go:117] "RemoveContainer" containerID="d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72"
Sep 15 21:44:33.500497 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.500331    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"
Sep 15 21:44:33.510413 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.510392    2551 scope.go:117] "RemoveContainer" containerID="d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72"
Sep 15 21:44:33.510696 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:44:33.510673    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72\": container with ID starting with d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72 not found: ID does not exist" containerID="d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72"
Sep 15 21:44:33.510769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.510704    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72"} err="failed to get container status \"d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72\": rpc error: code = NotFound desc = could not find container \"d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72\": container with ID starting with d4bd1cbb86eeeb2b8dd93bebe124efd95db33e7ff71c6306ea79238e2fbbde72 not found: ID does not exist"
Sep 15 21:44:33.526830 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.526795    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"]
Sep 15 21:44:33.535755 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.535733    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c0f45-9ff6dbf64-77vc5"]
Sep 15 21:44:33.769403 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:33.769323    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af86853-ae65-46bb-9155-2cee2a4ce17e" path="/var/lib/kubelet/pods/7af86853-ae65-46bb-9155-2cee2a4ce17e/volumes"
Sep 15 21:44:38.308748 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:38.308716    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"]
Sep 15 21:44:53.328483 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.328418    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb" podUID="eff64f60-5db7-400f-869f-5e19ff7e1ebd" containerName="main" containerID="cri-o://52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48" gracePeriod=45
Sep 15 21:44:53.582311 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.582247    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:44:53.604870 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.604833    2551 generic.go:356] "Generic (PLEG): container finished" podID="eff64f60-5db7-400f-869f-5e19ff7e1ebd" containerID="52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48" exitCode=0
Sep 15 21:44:53.605057 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.604880    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"
Sep 15 21:44:53.605057 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.604898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb" event={"ID":"eff64f60-5db7-400f-869f-5e19ff7e1ebd","Type":"ContainerDied","Data":"52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48"}
Sep 15 21:44:53.605057 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.604940    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb" event={"ID":"eff64f60-5db7-400f-869f-5e19ff7e1ebd","Type":"ContainerDied","Data":"7740d712756e033fe2926b47fb6b027d34534ba5850de23dfedf567048a80634"}
Sep 15 21:44:53.605057 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.604963    2551 scope.go:117] "RemoveContainer" containerID="52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48"
Sep 15 21:44:53.616393 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.616356    2551 scope.go:117] "RemoveContainer" containerID="52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48"
Sep 15 21:44:53.616776 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:44:53.616756    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48\": container with ID starting with 52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48 not found: ID does not exist" containerID="52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48"
Sep 15 21:44:53.616877 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.616784    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48"} err="failed to get container status \"52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48\": rpc error: code = NotFound desc = could not find container \"52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48\": container with ID starting with 52c81277de9eb5c43028a6acc86359c5f112a9b5dcffd1a56f10d36ff9deba48 not found: ID does not exist"
Sep 15 21:44:53.733489 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733460    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tmp-dir\") pod \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") "
Sep 15 21:44:53.733676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733497    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-model-cache\") pod \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") "
Sep 15 21:44:53.733676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733561    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-dshm\") pod \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") "
Sep 15 21:44:53.733676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733621    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-home\") pod \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") "
Sep 15 21:44:53.733676 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733652    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/eff64f60-5db7-400f-869f-5e19ff7e1ebd-kube-api-access-mb7xx\") pod \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") "
Sep 15 21:44:53.733913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733685    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tls-certs\") pod \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\" (UID: \"eff64f60-5db7-400f-869f-5e19ff7e1ebd\") "
Sep 15 21:44:53.733913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733775    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eff64f60-5db7-400f-869f-5e19ff7e1ebd" (UID: "eff64f60-5db7-400f-869f-5e19ff7e1ebd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:53.733913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733815    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-model-cache" (OuterVolumeSpecName: "model-cache") pod "eff64f60-5db7-400f-869f-5e19ff7e1ebd" (UID: "eff64f60-5db7-400f-869f-5e19ff7e1ebd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:53.734061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.733912    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-home" (OuterVolumeSpecName: "home") pod "eff64f60-5db7-400f-869f-5e19ff7e1ebd" (UID: "eff64f60-5db7-400f-869f-5e19ff7e1ebd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:53.734061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.734033    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:53.734061 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.734052    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:53.734222 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.734068    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:53.736125 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.736091    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-dshm" (OuterVolumeSpecName: "dshm") pod "eff64f60-5db7-400f-869f-5e19ff7e1ebd" (UID: "eff64f60-5db7-400f-869f-5e19ff7e1ebd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:53.736442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.736418    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff64f60-5db7-400f-869f-5e19ff7e1ebd-kube-api-access-mb7xx" (OuterVolumeSpecName: "kube-api-access-mb7xx") pod "eff64f60-5db7-400f-869f-5e19ff7e1ebd" (UID: "eff64f60-5db7-400f-869f-5e19ff7e1ebd"). InnerVolumeSpecName "kube-api-access-mb7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:53.736442 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.736424    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eff64f60-5db7-400f-869f-5e19ff7e1ebd" (UID: "eff64f60-5db7-400f-869f-5e19ff7e1ebd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:53.835638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.835533    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7xx\" (UniqueName: \"kubernetes.io/projected/eff64f60-5db7-400f-869f-5e19ff7e1ebd-kube-api-access-mb7xx\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:53.835638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.835584    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff64f60-5db7-400f-869f-5e19ff7e1ebd-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:53.835638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.835595    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff64f60-5db7-400f-869f-5e19ff7e1ebd-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:44:53.922108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.922074    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"]
Sep 15 21:44:53.924014 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:53.923990    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0f45/ipp-route-7c0f45-kserve-759b64d59f-n7vpb"]
Sep 15 21:44:55.768603 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:44:55.768574    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff64f60-5db7-400f-869f-5e19ff7e1ebd" path="/var/lib/kubelet/pods/eff64f60-5db7-400f-869f-5e19ff7e1ebd/volumes"
Sep 15 21:45:00.147964 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.147929    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"]
Sep 15 21:45:00.148519 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148498    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy"
Sep 15 21:45:00.148567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148523    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy"
Sep 15 21:45:00.148567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148546    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff64f60-5db7-400f-869f-5e19ff7e1ebd" containerName="main"
Sep 15 21:45:00.148567 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148555    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff64f60-5db7-400f-869f-5e19ff7e1ebd" containerName="main"
Sep 15 21:45:00.148675 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148568    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy"
Sep 15 21:45:00.148675 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148578    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy"
Sep 15 21:45:00.148675 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148599    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af86853-ae65-46bb-9155-2cee2a4ce17e" containerName="maas-api"
Sep 15 21:45:00.148675 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148604    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af86853-ae65-46bb-9155-2cee2a4ce17e" containerName="maas-api"
Sep 15 21:45:00.148675 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148666    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy"
Sep 15 21:45:00.148675 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148675    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af86853-ae65-46bb-9155-2cee2a4ce17e" containerName="maas-api"
Sep 15 21:45:00.148874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148681    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff64f60-5db7-400f-869f-5e19ff7e1ebd" containerName="main"
Sep 15 21:45:00.148874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.148687    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="667429f6-0017-472c-a27a-d5f38af01948" containerName="istio-proxy"
Sep 15 21:45:00.151126 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.151109    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"
Sep 15 21:45:00.154449 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.154425    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:45:00.157505 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.157469    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"]
Sep 15 21:45:00.164160 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.164125    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"]
Sep 15 21:45:00.164316 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.164248    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"
Sep 15 21:45:00.164781 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.164761    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"
Sep 15 21:45:00.179922 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.179894    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"]
Sep 15 21:45:00.192373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.192348    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"
Sep 15 21:45:00.443343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.443294    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"]
Sep 15 21:45:00.446136 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:45:00.446108    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19e4aa1_05eb_44ef_8d64_4cb28a6c156c.slice/crio-33e148fa38aa31c736c6ac4be532f28d11993cd5aa32f9b1602b246b8a04e34c WatchSource:0}: Error finding container 33e148fa38aa31c736c6ac4be532f28d11993cd5aa32f9b1602b246b8a04e34c: Status 404 returned error can't find the container with id 33e148fa38aa31c736c6ac4be532f28d11993cd5aa32f9b1602b246b8a04e34c
Sep 15 21:45:00.472796 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.472774    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"]
Sep 15 21:45:00.476264 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:45:00.476238    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce0fb9b_f28c_4eea_ae39_d50ec295cf75.slice/crio-c2bb5fceb4885b10329a87607e67085a9ecd7b548eee8914232f2dd5bbb41c98 WatchSource:0}: Error finding container c2bb5fceb4885b10329a87607e67085a9ecd7b548eee8914232f2dd5bbb41c98: Status 404 returned error can't find the container with id c2bb5fceb4885b10329a87607e67085a9ecd7b548eee8914232f2dd5bbb41c98
Sep 15 21:45:00.645861 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.645819    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2" event={"ID":"dce0fb9b-f28c-4eea-ae39-d50ec295cf75","Type":"ContainerStarted","Data":"c2bb5fceb4885b10329a87607e67085a9ecd7b548eee8914232f2dd5bbb41c98"}
Sep 15 21:45:00.647065 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:00.647045    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv" event={"ID":"f19e4aa1-05eb-44ef-8d64-4cb28a6c156c","Type":"ContainerStarted","Data":"33e148fa38aa31c736c6ac4be532f28d11993cd5aa32f9b1602b246b8a04e34c"}
Sep 15 21:45:01.653029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:01.652937    2551 generic.go:356] "Generic (PLEG): container finished" podID="dce0fb9b-f28c-4eea-ae39-d50ec295cf75" containerID="2351eba456f1e7530fe6bec20e04d69fdae1d24bfa6c4d3f14e0b00ebdfd42ba" exitCode=0
Sep 15 21:45:01.653426 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:01.653029    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2" event={"ID":"dce0fb9b-f28c-4eea-ae39-d50ec295cf75","Type":"ContainerDied","Data":"2351eba456f1e7530fe6bec20e04d69fdae1d24bfa6c4d3f14e0b00ebdfd42ba"}
Sep 15 21:45:01.654529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:01.654507    2551 generic.go:356] "Generic (PLEG): container finished" podID="f19e4aa1-05eb-44ef-8d64-4cb28a6c156c" containerID="3df658f269730933cce22682d9ca2164f1b9afa0a98e8b39064302e39771dc21" exitCode=0
Sep 15 21:45:01.654629 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:01.654539    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv" event={"ID":"f19e4aa1-05eb-44ef-8d64-4cb28a6c156c","Type":"ContainerDied","Data":"3df658f269730933cce22682d9ca2164f1b9afa0a98e8b39064302e39771dc21"}
Sep 15 21:45:02.804668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:02.804642    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"
Sep 15 21:45:02.807841 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:02.807823    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"
Sep 15 21:45:03.665656 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:03.665625    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"
Sep 15 21:45:03.665656 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:03.665632    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2" event={"ID":"dce0fb9b-f28c-4eea-ae39-d50ec295cf75","Type":"ContainerDied","Data":"c2bb5fceb4885b10329a87607e67085a9ecd7b548eee8914232f2dd5bbb41c98"}
Sep 15 21:45:03.665901 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:03.665683    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2bb5fceb4885b10329a87607e67085a9ecd7b548eee8914232f2dd5bbb41c98"
Sep 15 21:45:03.667015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:03.666995    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv"
Sep 15 21:45:03.667108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:03.667023    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-rswcv" event={"ID":"f19e4aa1-05eb-44ef-8d64-4cb28a6c156c","Type":"ContainerDied","Data":"33e148fa38aa31c736c6ac4be532f28d11993cd5aa32f9b1602b246b8a04e34c"}
Sep 15 21:45:03.667108 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:03.667070    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e148fa38aa31c736c6ac4be532f28d11993cd5aa32f9b1602b246b8a04e34c"
Sep 15 21:45:22.194674 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.194637    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"]
Sep 15 21:45:22.195062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.194901    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy" containerID="cri-o://a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a" gracePeriod=30
Sep 15 21:45:22.575844 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.575812    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"]
Sep 15 21:45:22.576185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.576172    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce0fb9b-f28c-4eea-ae39-d50ec295cf75" containerName="cleanup"
Sep 15 21:45:22.576251 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.576200    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce0fb9b-f28c-4eea-ae39-d50ec295cf75" containerName="cleanup"
Sep 15 21:45:22.576251 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.576221    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19e4aa1-05eb-44ef-8d64-4cb28a6c156c" containerName="cleanup"
Sep 15 21:45:22.576251 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.576228    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19e4aa1-05eb-44ef-8d64-4cb28a6c156c" containerName="cleanup"
Sep 15 21:45:22.576391 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.576280    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19e4aa1-05eb-44ef-8d64-4cb28a6c156c" containerName="cleanup"
Sep 15 21:45:22.576391 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.576288    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce0fb9b-f28c-4eea-ae39-d50ec295cf75" containerName="cleanup"
Sep 15 21:45:22.578950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.578934    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:22.581949 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.581926    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7fhv\""
Sep 15 21:45:22.589094 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.589065    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"]
Sep 15 21:45:22.722132 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.722092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9\" (UID: \"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:22.822944 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.822907    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9\" (UID: \"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:22.823571 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.823552    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9\" (UID: \"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:22.890950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:22.890863    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:23.165344 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:23.165320    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"]
Sep 15 21:45:23.167694 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:45:23.167667    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30cd91a_fdf0_4498_b48f_67bcf64ce6e0.slice/crio-f9b0114a20d251a0e86f2158a190743771dc8ff22c996c202566f380ff8ca98f WatchSource:0}: Error finding container f9b0114a20d251a0e86f2158a190743771dc8ff22c996c202566f380ff8ca98f: Status 404 returned error can't find the container with id f9b0114a20d251a0e86f2158a190743771dc8ff22c996c202566f380ff8ca98f
Sep 15 21:45:23.768609 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:23.768573    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9" event={"ID":"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0","Type":"ContainerStarted","Data":"1e00adf6f6cc4bfb89f8e704c9eda564d7d7c7513dee5aea4350b3d984a2f2dd"}
Sep 15 21:45:23.768973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:23.768637    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9" event={"ID":"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0","Type":"ContainerStarted","Data":"f9b0114a20d251a0e86f2158a190743771dc8ff22c996c202566f380ff8ca98f"}
Sep 15 21:45:23.785741 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:23.785694    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9" podStartSLOduration=1.7856790729999998 podStartE2EDuration="1.785679073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:23.782553091 +0000 UTC m=+2222.584422054" watchObservedRunningTime="2026-09-15 21:45:23.785679073 +0000 UTC m=+2222.587548016"
Sep 15 21:45:24.775702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:24.775665    2551 generic.go:356] "Generic (PLEG): container finished" podID="d30cd91a-fdf0-4498-b48f-67bcf64ce6e0" containerID="1e00adf6f6cc4bfb89f8e704c9eda564d7d7c7513dee5aea4350b3d984a2f2dd" exitCode=0
Sep 15 21:45:24.776173 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:24.775759    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9" event={"ID":"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0","Type":"ContainerDied","Data":"1e00adf6f6cc4bfb89f8e704c9eda564d7d7c7513dee5aea4350b3d984a2f2dd"}
Sep 15 21:45:25.924225 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:25.924174    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:26.058384 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:26.058298    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0-maas-api-service-ca\") pod \"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0\" (UID: \"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0\") "
Sep 15 21:45:26.058655 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:26.058632    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d30cd91a-fdf0-4498-b48f-67bcf64ce6e0" (UID: "d30cd91a-fdf0-4498-b48f-67bcf64ce6e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:26.159782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:26.159743    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0-maas-api-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:26.788165 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:26.788123    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"
Sep 15 21:45:26.788367 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:26.788168    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9" event={"ID":"d30cd91a-fdf0-4498-b48f-67bcf64ce6e0","Type":"ContainerDied","Data":"f9b0114a20d251a0e86f2158a190743771dc8ff22c996c202566f380ff8ca98f"}
Sep 15 21:45:26.788367 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:26.788241    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b0114a20d251a0e86f2158a190743771dc8ff22c996c202566f380ff8ca98f"
Sep 15 21:45:27.461782 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.461719    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:45:27.573110 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573072    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-credential-socket\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573129    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-podinfo\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573150    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-token\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573177    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfht7\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-kube-api-access-sfht7\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573327 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573258    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-socket\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573340    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-data\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573368    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-certs\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573394    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bd7bff-d9e4-43cf-92e9-baa95871105e-istiod-ca-cert\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573416    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-envoy\") pod \"19bd7bff-d9e4-43cf-92e9-baa95871105e\" (UID: \"19bd7bff-d9e4-43cf-92e9-baa95871105e\") "
Sep 15 21:45:27.573550 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573439    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:27.573821 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573604    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-data" (OuterVolumeSpecName: "istio-data") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:27.573821 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573625    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:27.573821 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573688    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bd7bff-d9e4-43cf-92e9-baa95871105e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:27.573821 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573809    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.574015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573827    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.574015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573840    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bd7bff-d9e4-43cf-92e9-baa95871105e-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.574015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573849    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.574015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.573980    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:27.575666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.575645    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-token" (OuterVolumeSpecName: "istio-token") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:27.575820 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.575806    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:27.576036 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.576007    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:45:27.576036 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.576014    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-kube-api-access-sfht7" (OuterVolumeSpecName: "kube-api-access-sfht7") pod "19bd7bff-d9e4-43cf-92e9-baa95871105e" (UID: "19bd7bff-d9e4-43cf-92e9-baa95871105e"). InnerVolumeSpecName "kube-api-access-sfht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:27.674516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.674481    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.674516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.674510    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.674516 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.674519    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.674757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.674528    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfht7\" (UniqueName: \"kubernetes.io/projected/19bd7bff-d9e4-43cf-92e9-baa95871105e-kube-api-access-sfht7\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.674757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.674538    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bd7bff-d9e4-43cf-92e9-baa95871105e-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:27.794500 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.794468    2551 generic.go:356] "Generic (PLEG): container finished" podID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerID="a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a" exitCode=0
Sep 15 21:45:27.794665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.794521    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"
Sep 15 21:45:27.794665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.794553    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" event={"ID":"19bd7bff-d9e4-43cf-92e9-baa95871105e","Type":"ContainerDied","Data":"a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a"}
Sep 15 21:45:27.794665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.794621    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5" event={"ID":"19bd7bff-d9e4-43cf-92e9-baa95871105e","Type":"ContainerDied","Data":"13f8146e9db0518797811cb6a5cdc48954b080c530f5f8fc7fea6482bd4509a1"}
Sep 15 21:45:27.794838 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.794650    2551 scope.go:117] "RemoveContainer" containerID="a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a"
Sep 15 21:45:27.805913 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.805887    2551 scope.go:117] "RemoveContainer" containerID="c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"
Sep 15 21:45:27.811277 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.811252    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"]
Sep 15 21:45:27.815363 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.815290    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c0f45-openshift-default-7db986b46d-c5zg5"]
Sep 15 21:45:27.815429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.815388    2551 scope.go:117] "RemoveContainer" containerID="a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a"
Sep 15 21:45:27.815706 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:45:27.815688    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a\": container with ID starting with a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a not found: ID does not exist" containerID="a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a"
Sep 15 21:45:27.815750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.815715    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a"} err="failed to get container status \"a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a\": rpc error: code = NotFound desc = could not find container \"a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a\": container with ID starting with a5e5cfd4459b6ca9f8247825dbc3a6d6d361ff2f2a480c870b7a8bd6f1d8e93a not found: ID does not exist"
Sep 15 21:45:27.815750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.815733    2551 scope.go:117] "RemoveContainer" containerID="c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"
Sep 15 21:45:27.815989 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:45:27.815973    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac\": container with ID starting with c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac not found: ID does not exist" containerID="c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"
Sep 15 21:45:27.816036 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:27.815994    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac"} err="failed to get container status \"c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac\": rpc error: code = NotFound desc = could not find container \"c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac\": container with ID starting with c70a711eb9f1583478a67cce85b7d1641d7d78e0d9a503e98d2a4545bc51abac not found: ID does not exist"
Sep 15 21:45:29.768677 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:29.768643    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" path="/var/lib/kubelet/pods/19bd7bff-d9e4-43cf-92e9-baa95871105e/volumes"
Sep 15 21:45:33.482790 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:33.482753    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"]
Sep 15 21:45:33.486406 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:33.486382    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e985b6-wths9"]
Sep 15 21:45:33.768455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:33.768375    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d30cd91a-fdf0-4498-b48f-67bcf64ce6e0" path="/var/lib/kubelet/pods/d30cd91a-fdf0-4498-b48f-67bcf64ce6e0/volumes"
Sep 15 21:45:40.276583 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:40.276551    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"]
Sep 15 21:45:40.276998 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:40.276775    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy" containerID="cri-o://0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951" gracePeriod=30
Sep 15 21:45:45.528021 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.527995    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:45:45.640502 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640409    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg7dl\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-kube-api-access-rg7dl\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640673 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640514    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-data\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640673 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640544    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istiod-ca-cert\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640673 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640595    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-envoy\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640673 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640627    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-certs\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640821    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-token\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640921 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640860    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-podinfo\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.640973 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640815    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:45.641033 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640981    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-credential-socket\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.641094 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641057    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-socket\") pod \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\" (UID: \"138592b7-88d3-4c81-be33-4ed6a8cdbd2b\") "
Sep 15 21:45:45.641094 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641063    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:45.641779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.640945    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:45.641779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641397    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:45.641779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641439    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:45.641779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641459    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.641779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641479    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.641779 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.641494    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.643363 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.643337    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:45:45.643451 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.643342    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:45.643579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.643552    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-kube-api-access-rg7dl" (OuterVolumeSpecName: "kube-api-access-rg7dl") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "kube-api-access-rg7dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:45.643674 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.643566    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "138592b7-88d3-4c81-be33-4ed6a8cdbd2b" (UID: "138592b7-88d3-4c81-be33-4ed6a8cdbd2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:45.742981 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.742947    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.742981 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.742977    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7dl\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-kube-api-access-rg7dl\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.742981 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.742987    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.743234 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.742995    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.743234 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.743003    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.743234 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.743011    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138592b7-88d3-4c81-be33-4ed6a8cdbd2b-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:45:45.884029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.883994    2551 generic.go:356] "Generic (PLEG): container finished" podID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerID="0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951" exitCode=0
Sep 15 21:45:45.884224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.884047    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"
Sep 15 21:45:45.884224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.884071    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" event={"ID":"138592b7-88d3-4c81-be33-4ed6a8cdbd2b","Type":"ContainerDied","Data":"0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951"}
Sep 15 21:45:45.884224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.884112    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r" event={"ID":"138592b7-88d3-4c81-be33-4ed6a8cdbd2b","Type":"ContainerDied","Data":"7a16b0fcacf88e850d0f86c75857d69f1b4f0755cd3ecdbc24fc8c8e47f99bf4"}
Sep 15 21:45:45.884224 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.884133    2551 scope.go:117] "RemoveContainer" containerID="0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951"
Sep 15 21:45:45.893026 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.893012    2551 scope.go:117] "RemoveContainer" containerID="b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"
Sep 15 21:45:45.900614 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.900589    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"]
Sep 15 21:45:45.901861 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.901848    2551 scope.go:117] "RemoveContainer" containerID="0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951"
Sep 15 21:45:45.902172 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:45:45.902151    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951\": container with ID starting with 0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951 not found: ID does not exist" containerID="0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951"
Sep 15 21:45:45.902223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.902181    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951"} err="failed to get container status \"0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951\": rpc error: code = NotFound desc = could not find container \"0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951\": container with ID starting with 0a8c204b94492bd588f1640c3efe03abb526192eca744ce427df44acdc34f951 not found: ID does not exist"
Sep 15 21:45:45.902223 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.902220    2551 scope.go:117] "RemoveContainer" containerID="b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"
Sep 15 21:45:45.902480 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:45:45.902462    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220\": container with ID starting with b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220 not found: ID does not exist" containerID="b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"
Sep 15 21:45:45.902530 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.902486    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220"} err="failed to get container status \"b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220\": rpc error: code = NotFound desc = could not find container \"b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220\": container with ID starting with b951be22347cff8ee8a07d24f066dc1abdeefdab7f024860960195b3d96d5220 not found: ID does not exist"
Sep 15 21:45:45.906166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:45.906142    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e985b6-openshift-default-6cdcf975db-5h52r"]
Sep 15 21:45:47.768733 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:47.768688    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" path="/var/lib/kubelet/pods/138592b7-88d3-4c81-be33-4ed6a8cdbd2b/volumes"
Sep 15 21:45:58.306530 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:58.306470    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"]
Sep 15 21:45:58.307134 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:45:58.306750    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy" containerID="cri-o://f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0" gracePeriod=30
Sep 15 21:46:03.561103 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.561078    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:46:03.710969 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.710933    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-socket\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.710998    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-data\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711033    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-token\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711058    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28668168-ee46-4e5b-8401-d61c6ba9076f-istiod-ca-cert\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711096    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-podinfo\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711130    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-credential-socket\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711166 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711156    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-certs\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711272    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfxq2\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-kube-api-access-bfxq2\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-envoy\") pod \"28668168-ee46-4e5b-8401-d61c6ba9076f\" (UID: \"28668168-ee46-4e5b-8401-d61c6ba9076f\") "
Sep 15 21:46:03.711523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711310    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:03.711523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711431    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-data" (OuterVolumeSpecName: "istio-data") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:03.711747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711541    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:03.711747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711467    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28668168-ee46-4e5b-8401-d61c6ba9076f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:03.711747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711613    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28668168-ee46-4e5b-8401-d61c6ba9076f-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.711747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711630    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.711747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711639    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.711747 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711648    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.712018 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.711772    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:03.713508 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.713479    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:46:03.713508 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.713481    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-token" (OuterVolumeSpecName: "istio-token") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:03.713670 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.713536    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-kube-api-access-bfxq2" (OuterVolumeSpecName: "kube-api-access-bfxq2") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "kube-api-access-bfxq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:03.713670 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.713650    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28668168-ee46-4e5b-8401-d61c6ba9076f" (UID: "28668168-ee46-4e5b-8401-d61c6ba9076f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:03.812296 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.812263    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.812296 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.812294    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.812296 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.812304    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.812511 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.812314    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxq2\" (UniqueName: \"kubernetes.io/projected/28668168-ee46-4e5b-8401-d61c6ba9076f-kube-api-access-bfxq2\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.812511 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.812325    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28668168-ee46-4e5b-8401-d61c6ba9076f-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:03.981769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.981677    2551 generic.go:356] "Generic (PLEG): container finished" podID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerID="f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0" exitCode=0
Sep 15 21:46:03.981769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.981726    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" event={"ID":"28668168-ee46-4e5b-8401-d61c6ba9076f","Type":"ContainerDied","Data":"f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0"}
Sep 15 21:46:03.981769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.981749    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"
Sep 15 21:46:03.981769 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.981772    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm" event={"ID":"28668168-ee46-4e5b-8401-d61c6ba9076f","Type":"ContainerDied","Data":"2a954264a1f6641fd1d4527e17340dc9b3fc6e449b5effd2feb14affc6f9e1c1"}
Sep 15 21:46:03.982100 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.981793    2551 scope.go:117] "RemoveContainer" containerID="f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0"
Sep 15 21:46:03.990462 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.990440    2551 scope.go:117] "RemoveContainer" containerID="1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"
Sep 15 21:46:03.998047 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.998021    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"]
Sep 15 21:46:03.999005 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.998980    2551 scope.go:117] "RemoveContainer" containerID="f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0"
Sep 15 21:46:03.999332 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:46:03.999305    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0\": container with ID starting with f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0 not found: ID does not exist" containerID="f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0"
Sep 15 21:46:03.999397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.999343    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0"} err="failed to get container status \"f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0\": rpc error: code = NotFound desc = could not find container \"f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0\": container with ID starting with f4cb752cc660deec46068afd19318a60adf06f96dd1e7a61bc2dafc5100ae2c0 not found: ID does not exist"
Sep 15 21:46:03.999397 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.999362    2551 scope.go:117] "RemoveContainer" containerID="1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"
Sep 15 21:46:03.999639 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:46:03.999622    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623\": container with ID starting with 1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623 not found: ID does not exist" containerID="1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"
Sep 15 21:46:03.999707 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:03.999644    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623"} err="failed to get container status \"1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623\": rpc error: code = NotFound desc = could not find container \"1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623\": container with ID starting with 1395fe98998ef1770886c7b92b3d8c159cb87c6be3e567a7b85e307c28924623 not found: ID does not exist"
Sep 15 21:46:04.002360 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:04.002340    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c4dc8-openshift-default-5d74896b78-zgjqm"]
Sep 15 21:46:05.768719 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:05.768684    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" path="/var/lib/kubelet/pods/28668168-ee46-4e5b-8401-d61c6ba9076f/volumes"
Sep 15 21:46:09.425450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:09.425419    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"]
Sep 15 21:46:09.427762 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:09.427737    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7c57b6-29825145-xwml2"]
Sep 15 21:46:09.769637 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:09.769559    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce0fb9b-f28c-4eea-ae39-d50ec295cf75" path="/var/lib/kubelet/pods/dce0fb9b-f28c-4eea-ae39-d50ec295cf75/volumes"
Sep 15 21:46:16.220580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:16.220543    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"]
Sep 15 21:46:16.220992 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:16.220808    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy" containerID="cri-o://faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d" gracePeriod=30
Sep 15 21:46:21.475712 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.475688    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:46:21.570818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.570734    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-token\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.570979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.570822    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-socket\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.570979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.570861    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-envoy\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.570979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.570917    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-data\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.570979 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.570963    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-podinfo\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.571226 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.570999    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-certs\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.571226 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571025    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52c3134-abb9-4192-bc05-23fc31e162b8-istiod-ca-cert\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.571226 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571133    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:21.571412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571259    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:21.571412 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571334    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:21.571517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571439    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-credential-socket\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.571517 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571482    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4xzb\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-kube-api-access-s4xzb\") pod \"d52c3134-abb9-4192-bc05-23fc31e162b8\" (UID: \"d52c3134-abb9-4192-bc05-23fc31e162b8\") "
Sep 15 21:46:21.571728 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571558    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52c3134-abb9-4192-bc05-23fc31e162b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:46:21.571845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571795    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-data\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.571845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571815    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.571845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571829    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52c3134-abb9-4192-bc05-23fc31e162b8-istiod-ca-cert\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.571845 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.571842    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-workload-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.572063 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.572012    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:21.573541 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.573511    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:21.573541 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.573520    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:21.573914 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.573890    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:46:21.573996 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.573912    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-kube-api-access-s4xzb" (OuterVolumeSpecName: "kube-api-access-s4xzb") pod "d52c3134-abb9-4192-bc05-23fc31e162b8" (UID: "d52c3134-abb9-4192-bc05-23fc31e162b8"). InnerVolumeSpecName "kube-api-access-s4xzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:21.672910 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.672863    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-podinfo\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.672910 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.672904    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-credential-socket\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.673178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.672920    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4xzb\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-kube-api-access-s4xzb\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.673178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.672935    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-token\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:21.673178 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:21.672948    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52c3134-abb9-4192-bc05-23fc31e162b8-istio-envoy\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:46:22.072674 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.072640    2551 generic.go:356] "Generic (PLEG): container finished" podID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerID="faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d" exitCode=0
Sep 15 21:46:22.072862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.072696    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"
Sep 15 21:46:22.072862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.072717    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" event={"ID":"d52c3134-abb9-4192-bc05-23fc31e162b8","Type":"ContainerDied","Data":"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"}
Sep 15 21:46:22.072862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.072771    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl" event={"ID":"d52c3134-abb9-4192-bc05-23fc31e162b8","Type":"ContainerDied","Data":"50215bcf927dfc6732a1e83f54e285fd3a55e4cf3887fc3684e238719011a0f5"}
Sep 15 21:46:22.072862 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.072794    2551 scope.go:117] "RemoveContainer" containerID="faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"
Sep 15 21:46:22.076609 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.076588    2551 scope.go:117] "RemoveContainer" containerID="9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"
Sep 15 21:46:22.084453 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.084435    2551 scope.go:117] "RemoveContainer" containerID="9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"
Sep 15 21:46:22.084874 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.084856    2551 scope.go:117] "RemoveContainer" containerID="faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"
Sep 15 21:46:22.085081 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:46:22.085057    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f\": container with ID starting with 9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f not found: ID does not exist" containerID="9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"
Sep 15 21:46:22.085141 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:46:22.085102    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d\": container with ID starting with faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d not found: ID does not exist" containerID="faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"
Sep 15 21:46:22.085141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.085105    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"} err="failed to get container status \"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f\": rpc error: code = NotFound desc = could not find container \"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f\": container with ID starting with 9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f not found: ID does not exist"
Sep 15 21:46:22.085141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.085124    2551 scope.go:117] "RemoveContainer" containerID="faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"
Sep 15 21:46:22.085315 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:46:22.085136    2551 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d\": rpc error: code = NotFound desc = could not find container \"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d\": container with ID starting with faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d not found: ID does not exist" containerID="faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"
Sep 15 21:46:22.085451 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.085431    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d"} err="failed to get container status \"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d\": rpc error: code = NotFound desc = could not find container \"faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d\": container with ID starting with faebc0e4e74b210ab194cb6b8c85064f92f4ef68f5602d2b14fe018a3d0fb63d not found: ID does not exist"
Sep 15 21:46:22.085505 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.085453    2551 scope.go:117] "RemoveContainer" containerID="9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"
Sep 15 21:46:22.085702 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.085677    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f"} err="failed to get container status \"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f\": rpc error: code = NotFound desc = could not find container \"9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f\": container with ID starting with 9d7eba28db7ba6eaffc003be61e779ef0846d53e12bb7494c8cdd719e78b8e1f not found: ID does not exist"
Sep 15 21:46:22.094335 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.094316    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"]
Sep 15 21:46:22.101437 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:22.101416    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7c57b6-openshift-default-679755fc5d-8kmfl"]
Sep 15 21:46:23.768083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:46:23.768049    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" path="/var/lib/kubelet/pods/d52c3134-abb9-4192-bc05-23fc31e162b8/volumes"
Sep 15 21:47:42.476208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476151    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65b5854d44-nchvl"]
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476490    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476502    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476510    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476515    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476524    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476529    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476536    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476541    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476549    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476554    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476560    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476564    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476574    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30cd91a-fdf0-4498-b48f-67bcf64ce6e0" containerName="revoke-keys"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476579    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30cd91a-fdf0-4498-b48f-67bcf64ce6e0" containerName="revoke-keys"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476584    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476589    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476634    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy"
Sep 15 21:47:42.476638 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476643    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy"
Sep 15 21:47:42.477211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476649    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy"
Sep 15 21:47:42.477211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476656    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy"
Sep 15 21:47:42.477211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476661    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52c3134-abb9-4192-bc05-23fc31e162b8" containerName="istio-proxy"
Sep 15 21:47:42.477211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476666    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bd7bff-d9e4-43cf-92e9-baa95871105e" containerName="istio-proxy"
Sep 15 21:47:42.477211 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.476671    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30cd91a-fdf0-4498-b48f-67bcf64ce6e0" containerName="revoke-keys"
Sep 15 21:47:42.479429 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.479409    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.481529 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.481509    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:47:42.482353 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.482333    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wvswp\""
Sep 15 21:47:42.482353 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.482351    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:47:42.482526 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.482359    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:47:42.488026 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.488001    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b5854d44-nchvl"]
Sep 15 21:47:42.575186 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.575140    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57cs\" (UniqueName: \"kubernetes.io/projected/be64d3f5-a094-4698-8ae1-aabe0dfda902-kube-api-access-p57cs\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.575425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.575319    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be64d3f5-a094-4698-8ae1-aabe0dfda902-metrics-certs\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.575425 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.575362    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be64d3f5-a094-4698-8ae1-aabe0dfda902-webhook-cert\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.675787 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.675753    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be64d3f5-a094-4698-8ae1-aabe0dfda902-metrics-certs\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.675787 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.675792    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be64d3f5-a094-4698-8ae1-aabe0dfda902-webhook-cert\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.676000 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.675817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57cs\" (UniqueName: \"kubernetes.io/projected/be64d3f5-a094-4698-8ae1-aabe0dfda902-kube-api-access-p57cs\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.678346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.678322    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be64d3f5-a094-4698-8ae1-aabe0dfda902-metrics-certs\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.678461 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.678426    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be64d3f5-a094-4698-8ae1-aabe0dfda902-webhook-cert\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.683464 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.683442    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57cs\" (UniqueName: \"kubernetes.io/projected/be64d3f5-a094-4698-8ae1-aabe0dfda902-kube-api-access-p57cs\") pod \"maas-controller-65b5854d44-nchvl\" (UID: \"be64d3f5-a094-4698-8ae1-aabe0dfda902\") " pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:42.791185 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:42.791096    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:43.087377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:43.087351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65b5854d44-nchvl"]
Sep 15 21:47:43.088677 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:47:43.088648    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe64d3f5_a094_4698_8ae1_aabe0dfda902.slice/crio-ac2963b95f5e2e18ece85da64b29c8219ea51d6db71ac504f4437f6127fbe5bb WatchSource:0}: Error finding container ac2963b95f5e2e18ece85da64b29c8219ea51d6db71ac504f4437f6127fbe5bb: Status 404 returned error can't find the container with id ac2963b95f5e2e18ece85da64b29c8219ea51d6db71ac504f4437f6127fbe5bb
Sep 15 21:47:43.089969 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:43.089952    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:47:43.445006 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:43.444975    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-nchvl" event={"ID":"be64d3f5-a094-4698-8ae1-aabe0dfda902","Type":"ContainerStarted","Data":"ac2963b95f5e2e18ece85da64b29c8219ea51d6db71ac504f4437f6127fbe5bb"}
Sep 15 21:47:44.450854 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:44.450817    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65b5854d44-nchvl" event={"ID":"be64d3f5-a094-4698-8ae1-aabe0dfda902","Type":"ContainerStarted","Data":"014cd11fa3a912c118575ae07cf781a8a83d1de91c892e03f9a98f2a7e927e80"}
Sep 15 21:47:44.451292 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:44.450957    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:47:44.468179 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:44.468131    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65b5854d44-nchvl" podStartSLOduration=2.066841221 podStartE2EDuration="2.468110988s" totalImagesPullingTime="401.269767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:44.465525127 +0000 UTC m=+2363.267394069" watchObservedRunningTime="2026-09-15 21:47:44.468110988 +0000 UTC m=+2363.269979930"
Sep 15 21:47:55.457728 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:47:55.457700    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65b5854d44-nchvl"
Sep 15 21:50:52.333287 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.333256    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"]
Sep 15 21:50:52.333713 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.333596    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy"
Sep 15 21:50:52.333713 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.333606    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy"
Sep 15 21:50:52.333713 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.333662    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="138592b7-88d3-4c81-be33-4ed6a8cdbd2b" containerName="istio-proxy"
Sep 15 21:50:52.333713 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.333669    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="28668168-ee46-4e5b-8401-d61c6ba9076f" containerName="istio-proxy"
Sep 15 21:50:52.336574 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.336555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.338864 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.338841    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b6f6c259-kserve-self-signed-certs\""
Sep 15 21:50:52.348080 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.348058    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"]
Sep 15 21:50:52.491375 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.491331    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pqh\" (UniqueName: \"kubernetes.io/projected/4b393668-53e6-4345-b9e6-d7194860e5ce-kube-api-access-x5pqh\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.491585 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.491441    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b393668-53e6-4345-b9e6-d7194860e5ce-tls-certs\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.491585 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.491527    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-dshm\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.491727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.491594    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-model-cache\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.491727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.491651    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-home\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.491727 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.491693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-tmp-dir\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.592937    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-model-cache\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.592979    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-home\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593029 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593008    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-tmp-dir\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593046    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pqh\" (UniqueName: \"kubernetes.io/projected/4b393668-53e6-4345-b9e6-d7194860e5ce-kube-api-access-x5pqh\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593079    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b393668-53e6-4345-b9e6-d7194860e5ce-tls-certs\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593346 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593113    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-dshm\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593510 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593402    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-home\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593510 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593439    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-model-cache\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.593621 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.593561    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-tmp-dir\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.595494 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.595476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-dshm\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.595806 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.595788    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b393668-53e6-4345-b9e6-d7194860e5ce-tls-certs\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.604956 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.604935    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pqh\" (UniqueName: \"kubernetes.io/projected/4b393668-53e6-4345-b9e6-d7194860e5ce-kube-api-access-x5pqh\") pod \"e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") " pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.648535 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.648497    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:50:52.953159 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:52.953132    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"]
Sep 15 21:50:52.955617 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:50:52.955586    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b393668_53e6_4345_b9e6_d7194860e5ce.slice/crio-0828a4035571002e8294913930db3fcd0f082339b5ca328bdc8e8e222666ebc3 WatchSource:0}: Error finding container 0828a4035571002e8294913930db3fcd0f082339b5ca328bdc8e8e222666ebc3: Status 404 returned error can't find the container with id 0828a4035571002e8294913930db3fcd0f082339b5ca328bdc8e8e222666ebc3
Sep 15 21:50:53.335972 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:53.335889    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc" event={"ID":"4b393668-53e6-4345-b9e6-d7194860e5ce","Type":"ContainerStarted","Data":"40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68"}
Sep 15 21:50:53.335972 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:53.335932    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc" event={"ID":"4b393668-53e6-4345-b9e6-d7194860e5ce","Type":"ContainerStarted","Data":"0828a4035571002e8294913930db3fcd0f082339b5ca328bdc8e8e222666ebc3"}
Sep 15 21:50:53.353704 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:50:53.353653    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc" podStartSLOduration=1.353637929 podStartE2EDuration="1.353637929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:50:53.35258769 +0000 UTC m=+2552.154456629" watchObservedRunningTime="2026-09-15 21:50:53.353637929 +0000 UTC m=+2552.155506872"
Sep 15 21:51:02.649279 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:02.649225    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:51:02.649925 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:02.649298    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:51:02.661985 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:02.661956    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:51:03.393051 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:03.393018    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:51:22.133141 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:22.133112    2551 scope.go:117] "RemoveContainer" containerID="2351eba456f1e7530fe6bec20e04d69fdae1d24bfa6c4d3f14e0b00ebdfd42ba"
Sep 15 21:51:22.614297 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:22.614266    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"]
Sep 15 21:51:37.634892 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.634820    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc" podUID="4b393668-53e6-4345-b9e6-d7194860e5ce" containerName="main" containerID="cri-o://40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68" gracePeriod=45
Sep 15 21:51:37.896792 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.896725    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:51:37.922444 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922416    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-model-cache\") pod \"4b393668-53e6-4345-b9e6-d7194860e5ce\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") "
Sep 15 21:51:37.922626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922455    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5pqh\" (UniqueName: \"kubernetes.io/projected/4b393668-53e6-4345-b9e6-d7194860e5ce-kube-api-access-x5pqh\") pod \"4b393668-53e6-4345-b9e6-d7194860e5ce\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") "
Sep 15 21:51:37.922626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922493    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-home\") pod \"4b393668-53e6-4345-b9e6-d7194860e5ce\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") "
Sep 15 21:51:37.922626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922530    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-tmp-dir\") pod \"4b393668-53e6-4345-b9e6-d7194860e5ce\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") "
Sep 15 21:51:37.922626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922570    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b393668-53e6-4345-b9e6-d7194860e5ce-tls-certs\") pod \"4b393668-53e6-4345-b9e6-d7194860e5ce\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") "
Sep 15 21:51:37.922626 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922592    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-dshm\") pod \"4b393668-53e6-4345-b9e6-d7194860e5ce\" (UID: \"4b393668-53e6-4345-b9e6-d7194860e5ce\") "
Sep 15 21:51:37.922930 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922707    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b393668-53e6-4345-b9e6-d7194860e5ce" (UID: "4b393668-53e6-4345-b9e6-d7194860e5ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:37.922930 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922815    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-home" (OuterVolumeSpecName: "home") pod "4b393668-53e6-4345-b9e6-d7194860e5ce" (UID: "4b393668-53e6-4345-b9e6-d7194860e5ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:37.922930 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.922830    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b393668-53e6-4345-b9e6-d7194860e5ce" (UID: "4b393668-53e6-4345-b9e6-d7194860e5ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:37.923124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.923008    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-model-cache\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:37.923124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.923027    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-home\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:37.923124 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.923042    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-tmp-dir\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:37.924884 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.924851    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-dshm" (OuterVolumeSpecName: "dshm") pod "4b393668-53e6-4345-b9e6-d7194860e5ce" (UID: "4b393668-53e6-4345-b9e6-d7194860e5ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:37.925023 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.924892    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b393668-53e6-4345-b9e6-d7194860e5ce-kube-api-access-x5pqh" (OuterVolumeSpecName: "kube-api-access-x5pqh") pod "4b393668-53e6-4345-b9e6-d7194860e5ce" (UID: "4b393668-53e6-4345-b9e6-d7194860e5ce"). InnerVolumeSpecName "kube-api-access-x5pqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:37.925023 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:37.924965    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b393668-53e6-4345-b9e6-d7194860e5ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b393668-53e6-4345-b9e6-d7194860e5ce" (UID: "4b393668-53e6-4345-b9e6-d7194860e5ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:38.023666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.023633    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b393668-53e6-4345-b9e6-d7194860e5ce-tls-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:38.023666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.023660    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b393668-53e6-4345-b9e6-d7194860e5ce-dshm\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:38.023666 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.023670    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pqh\" (UniqueName: \"kubernetes.io/projected/4b393668-53e6-4345-b9e6-d7194860e5ce-kube-api-access-x5pqh\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:38.548858 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.548820    2551 generic.go:356] "Generic (PLEG): container finished" podID="4b393668-53e6-4345-b9e6-d7194860e5ce" containerID="40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68" exitCode=0
Sep 15 21:51:38.549028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.548927    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"
Sep 15 21:51:38.549028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.548951    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc" event={"ID":"4b393668-53e6-4345-b9e6-d7194860e5ce","Type":"ContainerDied","Data":"40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68"}
Sep 15 21:51:38.549028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.548989    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc" event={"ID":"4b393668-53e6-4345-b9e6-d7194860e5ce","Type":"ContainerDied","Data":"0828a4035571002e8294913930db3fcd0f082339b5ca328bdc8e8e222666ebc3"}
Sep 15 21:51:38.549028 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.549010    2551 scope.go:117] "RemoveContainer" containerID="40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68"
Sep 15 21:51:38.558313 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.558252    2551 scope.go:117] "RemoveContainer" containerID="40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68"
Sep 15 21:51:38.558558 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:51:38.558539    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68\": container with ID starting with 40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68 not found: ID does not exist" containerID="40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68"
Sep 15 21:51:38.558598 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.558568    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68"} err="failed to get container status \"40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68\": rpc error: code = NotFound desc = could not find container \"40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68\": container with ID starting with 40f51dae96cb0b331ab1f87e9866667e3a064b0664ce97d6b7d04e63e7f1fa68 not found: ID does not exist"
Sep 15 21:51:38.571537 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.571512    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"]
Sep 15 21:51:38.576310 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:38.576289    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b6f6c259-kserve-8fd854755-rgfsc"]
Sep 15 21:51:39.768523 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:39.768488    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b393668-53e6-4345-b9e6-d7194860e5ce" path="/var/lib/kubelet/pods/4b393668-53e6-4345-b9e6-d7194860e5ce/volumes"
Sep 15 21:51:41.773377 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.773348    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"]
Sep 15 21:51:41.773750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.773676    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b393668-53e6-4345-b9e6-d7194860e5ce" containerName="main"
Sep 15 21:51:41.773750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.773687    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b393668-53e6-4345-b9e6-d7194860e5ce" containerName="main"
Sep 15 21:51:41.773750 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.773741    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b393668-53e6-4345-b9e6-d7194860e5ce" containerName="main"
Sep 15 21:51:41.777927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.777904    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.788659 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.788632    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"]
Sep 15 21:51:41.849269 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.849229    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f4be330-9f4f-486f-9baa-3c891948022d-metrics-certs\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.849455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.849279    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f4be330-9f4f-486f-9baa-3c891948022d-maas-api-tls\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.849455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.849349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4be330-9f4f-486f-9baa-3c891948022d-openshift-service-ca\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.849455 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.849396    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8wj\" (UniqueName: \"kubernetes.io/projected/7f4be330-9f4f-486f-9baa-3c891948022d-kube-api-access-7r8wj\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.949851 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.949818    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f4be330-9f4f-486f-9baa-3c891948022d-metrics-certs\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.949851 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.949852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f4be330-9f4f-486f-9baa-3c891948022d-maas-api-tls\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.950083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.949984    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4be330-9f4f-486f-9baa-3c891948022d-openshift-service-ca\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.950083 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.950042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8wj\" (UniqueName: \"kubernetes.io/projected/7f4be330-9f4f-486f-9baa-3c891948022d-kube-api-access-7r8wj\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.950610 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.950585    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4be330-9f4f-486f-9baa-3c891948022d-openshift-service-ca\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.952409 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.952382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f4be330-9f4f-486f-9baa-3c891948022d-maas-api-tls\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.952504 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.952415    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f4be330-9f4f-486f-9baa-3c891948022d-metrics-certs\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:41.957786 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:41.957768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8wj\" (UniqueName: \"kubernetes.io/projected/7f4be330-9f4f-486f-9baa-3c891948022d-kube-api-access-7r8wj\") pod \"maas-api-74dc4c8bc5-xv5kk\" (UID: \"7f4be330-9f4f-486f-9baa-3c891948022d\") " pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:42.089823 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:42.089724    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:42.387705 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:42.387610    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"]
Sep 15 21:51:42.389158 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:51:42.389128    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4be330_9f4f_486f_9baa_3c891948022d.slice/crio-4d16fa9f729b1ddb0d18d9755a8fceff404232e2d8841809c9b886b8e4d878e6 WatchSource:0}: Error finding container 4d16fa9f729b1ddb0d18d9755a8fceff404232e2d8841809c9b886b8e4d878e6: Status 404 returned error can't find the container with id 4d16fa9f729b1ddb0d18d9755a8fceff404232e2d8841809c9b886b8e4d878e6
Sep 15 21:51:42.568818 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:42.568785    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk" event={"ID":"7f4be330-9f4f-486f-9baa-3c891948022d","Type":"ContainerStarted","Data":"4d16fa9f729b1ddb0d18d9755a8fceff404232e2d8841809c9b886b8e4d878e6"}
Sep 15 21:51:43.574757 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:43.574720    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk" event={"ID":"7f4be330-9f4f-486f-9baa-3c891948022d","Type":"ContainerStarted","Data":"12434b78fbebd624a90dc09e560036637c84647982daa5daa70f3d8173bbe83d"}
Sep 15 21:51:43.575249 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:43.574836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:43.596476 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:43.596428    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk" podStartSLOduration=1.873935506 podStartE2EDuration="2.59641336s" totalImagesPullingTime="722.477854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:51:43.594254614 +0000 UTC m=+2602.396123568" watchObservedRunningTime="2026-09-15 21:51:43.59641336 +0000 UTC m=+2602.398282305"
Sep 15 21:51:49.583947 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:49.583919    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74dc4c8bc5-xv5kk"
Sep 15 21:51:57.717328 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:57.717288    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"]
Sep 15 21:51:57.718565 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:57.717927    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" podUID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" containerName="maas-api" containerID="cri-o://f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9" gracePeriod=30
Sep 15 21:51:58.541112 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:58.541066    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" podUID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.31:8443/health\": dial tcp 10.133.0.31:8443: connect: connection refused"
Sep 15 21:51:58.966738 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:58.966715    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:51:59.000928 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.000446    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-openshift-service-ca\") pod \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") "
Sep 15 21:51:59.000928 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.000495    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs\") pod \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") "
Sep 15 21:51:59.000928 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.000556    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg8kw\" (UniqueName: \"kubernetes.io/projected/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-kube-api-access-rg8kw\") pod \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") "
Sep 15 21:51:59.000928 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.000640    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls\") pod \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\" (UID: \"7dd7a051-941c-42ef-b1fa-cb4915f75b5f\") "
Sep 15 21:51:59.001539 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.001492    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dd7a051-941c-42ef-b1fa-cb4915f75b5f" (UID: "7dd7a051-941c-42ef-b1fa-cb4915f75b5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:51:59.003562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.003495    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7dd7a051-941c-42ef-b1fa-cb4915f75b5f" (UID: "7dd7a051-941c-42ef-b1fa-cb4915f75b5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:59.003562 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.003524    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-kube-api-access-rg8kw" (OuterVolumeSpecName: "kube-api-access-rg8kw") pod "7dd7a051-941c-42ef-b1fa-cb4915f75b5f" (UID: "7dd7a051-941c-42ef-b1fa-cb4915f75b5f"). InnerVolumeSpecName "kube-api-access-rg8kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:59.004098 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.004070    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dd7a051-941c-42ef-b1fa-cb4915f75b5f" (UID: "7dd7a051-941c-42ef-b1fa-cb4915f75b5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:59.102208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.102095    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-maas-api-tls\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:59.102208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.102129    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-openshift-service-ca\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:59.102208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.102144    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-metrics-certs\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:59.102208 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.102157    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg8kw\" (UniqueName: \"kubernetes.io/projected/7dd7a051-941c-42ef-b1fa-cb4915f75b5f-kube-api-access-rg8kw\") on node \"ip-10-0-141-188.ec2.internal\" DevicePath \"\""
Sep 15 21:51:59.650447 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.650411    2551 generic.go:356] "Generic (PLEG): container finished" podID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" containerID="f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9" exitCode=0
Sep 15 21:51:59.650665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.650459    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"
Sep 15 21:51:59.650665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.650497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" event={"ID":"7dd7a051-941c-42ef-b1fa-cb4915f75b5f","Type":"ContainerDied","Data":"f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9"}
Sep 15 21:51:59.650665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.650562    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq" event={"ID":"7dd7a051-941c-42ef-b1fa-cb4915f75b5f","Type":"ContainerDied","Data":"094a3a5f6a333d59bb431c8a18825fba01bf5e6cf10cfd9639a1e22e0f3ea2bf"}
Sep 15 21:51:59.650665 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.650594    2551 scope.go:117] "RemoveContainer" containerID="f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9"
Sep 15 21:51:59.659950 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.659934    2551 scope.go:117] "RemoveContainer" containerID="f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9"
Sep 15 21:51:59.660276 ip-10-0-141-188 kubenswrapper[2551]: E0915 21:51:59.660254    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9\": container with ID starting with f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9 not found: ID does not exist" containerID="f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9"
Sep 15 21:51:59.660366 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.660286    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9"} err="failed to get container status \"f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9\": rpc error: code = NotFound desc = could not find container \"f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9\": container with ID starting with f7c4625b03e26de3a55e80752aa7127d5189229114c42ca6b66b63fa3f1efcc9 not found: ID does not exist"
Sep 15 21:51:59.671870 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.671841    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"]
Sep 15 21:51:59.676266 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.676245    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c7fc967-x7vnq"]
Sep 15 21:51:59.768744 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:51:59.768713    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" path="/var/lib/kubelet/pods/7dd7a051-941c-42ef-b1fa-cb4915f75b5f/volumes"
Sep 15 21:52:15.396966 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:15.396939    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8pzvp_5ede3f22-faa4-43fa-b430-4c37a8c417eb/manager/0.log"
Sep 15 21:52:15.505761 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:15.505728    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-g2lrz_695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0/manager/0.log"
Sep 15 21:52:15.629646 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:15.629613    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b5854d44-nchvl_be64d3f5-a094-4698-8ae1-aabe0dfda902/manager/0.log"
Sep 15 21:52:15.736156 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:15.736126    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k7t5c_01370159-7c02-4486-8ded-4133a68fd3ea/server/0.log"
Sep 15 21:52:15.850579 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:15.850546    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c89ng_cde5ef23-1fef-4673-b95c-27ef0a27575f/manager/0.log"
Sep 15 21:52:16.089999 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:16.089922    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-r56ql_888d57b2-9405-4496-9bc6-b5c3701aac96/manager/0.log"
Sep 15 21:52:16.098580 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:16.098556    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-r56ql_888d57b2-9405-4496-9bc6-b5c3701aac96/copy-manifests/0.log"
Sep 15 21:52:16.685484 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:16.685455    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-74dc4c8bc5-xv5kk_7f4be330-9f4f-486f-9baa-3c891948022d/maas-api/0.log"
Sep 15 21:52:16.790669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:16.790638    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825130-282nr_be5c797a-f974-4c79-9d46-3f662969df21/cleanup/0.log"
Sep 15 21:52:16.899856 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:16.899820    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825145-rswcv_f19e4aa1-05eb-44ef-8d64-4cb28a6c156c/cleanup/0.log"
Sep 15 21:52:17.010450 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:17.010364    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9dths_362d8645-020f-4cc0-b9d5-1f7376c6457e/postgres/0.log"
Sep 15 21:52:17.990927 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:17.990898    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bqxrq_95627f6b-0ef0-4535-b895-492dbcc68f2f/manager/0.log"
Sep 15 21:52:18.823734 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:18.823702    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq_059792c3-8fc3-45bd-9977-59c02ca5818e/istio-proxy/0.log"
Sep 15 21:52:19.303345 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:19.303316    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/1.log"
Sep 15 21:52:19.984772 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:19.984746    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vbnvd_15ae6ce1-b938-4232-90ac-e0d8118f4666/main/0.log"
Sep 15 21:52:20.201878 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:20.201851    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-kngfr_aeb2fe73-2ac3-4a9e-8600-7998fa8fe1f2/main/0.log"
Sep 15 21:52:20.425033 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:20.425000    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f97rmb_cc09a101-fc96-4a0e-8777-88d9a61f3bf9/main/0.log"
Sep 15 21:52:22.152186 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:22.152156    2551 scope.go:117] "RemoveContainer" containerID="1e00adf6f6cc4bfb89f8e704c9eda564d7d7c7513dee5aea4350b3d984a2f2dd"
Sep 15 21:52:28.883608 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:28.883573    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t22b7_c1e92fdc-eaa5-48fa-9584-3eb7d009467c/global-pull-secret-syncer/0.log"
Sep 15 21:52:29.018855 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:29.018826    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nzr6t_02ddff51-d8f4-4565-9e73-2d6796a09b90/konnectivity-agent/0.log"
Sep 15 21:52:29.133542 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:29.133517    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-188.ec2.internal_7dab564a33b33063370e62af110a6955/haproxy/0.log"
Sep 15 21:52:33.629669 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:33.629640    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bqxrq_95627f6b-0ef0-4535-b895-492dbcc68f2f/manager/0.log"
Sep 15 21:52:36.177685 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:36.177656    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6xhr_556a89df-43a5-4f4c-95af-5dda70c10295/node-exporter/0.log"
Sep 15 21:52:36.202808 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:36.202781    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6xhr_556a89df-43a5-4f4c-95af-5dda70c10295/kube-rbac-proxy/0.log"
Sep 15 21:52:36.229219 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:36.229179    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6xhr_556a89df-43a5-4f4c-95af-5dda70c10295/init-textfile/0.log"
Sep 15 21:52:38.694343 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.694059    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"]
Sep 15 21:52:38.697386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.695638    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" containerName="maas-api"
Sep 15 21:52:38.697386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.695669    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" containerName="maas-api"
Sep 15 21:52:38.697386 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.695822    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd7a051-941c-42ef-b1fa-cb4915f75b5f" containerName="maas-api"
Sep 15 21:52:38.701098 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.701074    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.703668 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.703649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sggc6\"/\"default-dockercfg-zzzv5\""
Sep 15 21:52:38.703781 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.703647    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggc6\"/\"kube-root-ca.crt\""
Sep 15 21:52:38.704729 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.704710    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggc6\"/\"openshift-service-ca.crt\""
Sep 15 21:52:38.707868 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.707846    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"]
Sep 15 21:52:38.746628 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.746596    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgnsn\" (UniqueName: \"kubernetes.io/projected/6abace35-549b-4679-a9fd-3d3273e8828f-kube-api-access-hgnsn\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.746628 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.746638    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-sys\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.746831 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.746754    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-lib-modules\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.746831 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.746788    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-proc\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.746906 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.746893    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-podres\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.847883 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.847850    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-podres\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.847888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgnsn\" (UniqueName: \"kubernetes.io/projected/6abace35-549b-4679-a9fd-3d3273e8828f-kube-api-access-hgnsn\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.847917    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-sys\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.847983    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-lib-modules\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.848012    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-proc\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848062 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.848044    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-podres\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848313 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.848063    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-sys\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848313 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.848100    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-proc\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.848313 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.848124    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6abace35-549b-4679-a9fd-3d3273e8828f-lib-modules\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:38.856640 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:38.856621    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgnsn\" (UniqueName: \"kubernetes.io/projected/6abace35-549b-4679-a9fd-3d3273e8828f-kube-api-access-hgnsn\") pod \"perf-node-gather-daemonset-sjqzn\" (UID: \"6abace35-549b-4679-a9fd-3d3273e8828f\") " pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:39.012373 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:39.012283    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:39.292266 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:39.292239    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"]
Sep 15 21:52:39.294507 ip-10-0-141-188 kubenswrapper[2551]: W0915 21:52:39.294478    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6abace35_549b_4679_a9fd_3d3273e8828f.slice/crio-e6f06a9af6567746b2dcaedd39e11667fce8f3139766567a32a4fdd86abe2e14 WatchSource:0}: Error finding container e6f06a9af6567746b2dcaedd39e11667fce8f3139766567a32a4fdd86abe2e14: Status 404 returned error can't find the container with id e6f06a9af6567746b2dcaedd39e11667fce8f3139766567a32a4fdd86abe2e14
Sep 15 21:52:39.836793 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:39.836756    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn" event={"ID":"6abace35-549b-4679-a9fd-3d3273e8828f","Type":"ContainerStarted","Data":"d8aec2034dc7042e7a74697b0321d7b20f3a5fa43f7243fb6e77e57f72bb80f6"}
Sep 15 21:52:39.837216 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:39.836808    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn" event={"ID":"6abace35-549b-4679-a9fd-3d3273e8828f","Type":"ContainerStarted","Data":"e6f06a9af6567746b2dcaedd39e11667fce8f3139766567a32a4fdd86abe2e14"}
Sep 15 21:52:39.837216 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:39.836841    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:39.853368 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:39.853148    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn" podStartSLOduration=1.853129657 podStartE2EDuration="1.853129657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:39.853005459 +0000 UTC m=+2658.654874399" watchObservedRunningTime="2026-09-15 21:52:39.853129657 +0000 UTC m=+2658.654998601"
Sep 15 21:52:41.355015 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:41.354985    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-42rkm_e0f0f048-6751-4841-abd2-323b67ae5e30/dns/0.log"
Sep 15 21:52:41.375273 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:41.375243    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-42rkm_e0f0f048-6751-4841-abd2-323b67ae5e30/kube-rbac-proxy/0.log"
Sep 15 21:52:41.517731 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:41.517704    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjwpf_05ad11b2-75b4-4a23-91cd-33be656bd8f6/dns-node-resolver/0.log"
Sep 15 21:52:42.050532 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:42.050504    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f4c7f9967-x6zfj_a8ec35e3-4232-4790-bd19-ab4801c3176a/registry/0.log"
Sep 15 21:52:42.118811 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:42.118788    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p2hgv_7b9309ce-ed18-4f88-8b8e-28074c47e3f2/node-ca/0.log"
Sep 15 21:52:43.039210 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:43.039177    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq_059792c3-8fc3-45bd-9977-59c02ca5818e/istio-proxy/0.log"
Sep 15 21:52:43.975337 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:43.975302    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/0.log"
Sep 15 21:52:43.978920 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:43.978890    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/1.log"
Sep 15 21:52:45.001599 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:45.001570    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6bhh_7165655a-da5a-4d68-af9e-0ffbfa8589b3/serve-healthcheck-canary/0.log"
Sep 15 21:52:45.710300 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:45.710272    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4zcm_5ffde2dd-c294-46ad-940f-9815514bab78/kube-rbac-proxy/0.log"
Sep 15 21:52:45.732505 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:45.732483    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4zcm_5ffde2dd-c294-46ad-940f-9815514bab78/exporter/0.log"
Sep 15 21:52:45.753977 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:45.753952    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q4zcm_5ffde2dd-c294-46ad-940f-9815514bab78/extractor/0.log"
Sep 15 21:52:45.851570 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:45.851544    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sggc6/perf-node-gather-daemonset-sjqzn"
Sep 15 21:52:48.153489 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.153464    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8pzvp_5ede3f22-faa4-43fa-b430-4c37a8c417eb/manager/0.log"
Sep 15 21:52:48.414773 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.414714    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq_059792c3-8fc3-45bd-9977-59c02ca5818e/istio-proxy/0.log"
Sep 15 21:52:48.430139 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.430105    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-g2lrz_695dfc64-b0f6-47ea-91e7-deb2f0f4f8c0/manager/0.log"
Sep 15 21:52:48.478056 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.478032    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65b5854d44-nchvl_be64d3f5-a094-4698-8ae1-aabe0dfda902/manager/0.log"
Sep 15 21:52:48.506788 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.506763    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k7t5c_01370159-7c02-4486-8ded-4133a68fd3ea/server/0.log"
Sep 15 21:52:48.595745 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.595716    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c89ng_cde5ef23-1fef-4673-b95c-27ef0a27575f/manager/0.log"
Sep 15 21:52:48.700314 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.700284    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-r56ql_888d57b2-9405-4496-9bc6-b5c3701aac96/manager/0.log"
Sep 15 21:52:48.723118 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:48.723097    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546ffd48db-r56ql_888d57b2-9405-4496-9bc6-b5c3701aac96/copy-manifests/0.log"
Sep 15 21:52:49.351004 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:49.350972    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/0.log"
Sep 15 21:52:49.353555 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:49.353534    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/1.log"
Sep 15 21:52:50.100411 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:50.100317    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68888cf94f-bjxlt_895c60fb-96d3-4ce1-ac49-2a7b102999c1/manager/0.log"
Sep 15 21:52:50.915926 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:50.915901    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vgkmq_059792c3-8fc3-45bd-9977-59c02ca5818e/istio-proxy/0.log"
Sep 15 21:52:51.890110 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:51.890079    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/0.log"
Sep 15 21:52:51.892692 ip-10-0-141-188 kubenswrapper[2551]: I0915 21:52:51.892673    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-8xm2l_7aee3b2d-7f62-48a5-89cf-e0d93904115f/istio-proxy/1.log"